{'fraction_non_alphanumeric': 0.021710649447875725, 'fraction_numerical': 0.00852621290577495, 'mean_word_length': 4.762492510485321, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Combining Observational and Experimental Data Using First-stage Covariates

Experiments are the gold standard for causal identification but often have limited scale, while observational datasets are large but often violate standard identification assumptions. To improve the estimation efficiency, we propose a new method that combines experimental and observational data by leveraging covariates that satisfy the first-stage relevance condition in the observational data. We first use the observational data to derive a biased estimate of the causal effect and then correct this bias using the experimental data. We show that this bias-corrected estimator is uncorrelated with the benchmark estimator that only uses the experimental data, and can thus be combined with the benchmark estimator to improve efficiency. Under a common experimental design that randomly assigns units into the experimental group, our method can reduce the variance by up to 50%, so that only half of the experimental sample is required to attain the same accuracy. This accuracy can be further improved if the experimental design accounts for this relevant first-stage covariate and select the experimental group differently.


Introduction
Measuring the causal effect of different business practices has become increasingly important for firms that have shifted their businesses online. Without understanding the causal effects of marketing mix variables such as prices, ads, and recommendations on outcomes such as clicks and purchases for different consumers, firms cannot optimize their algorithms and customize their marketing strategies. One popular solution is to use observational methods that often train a predictive model of consumer outcomes using detailed observational data of online browsing behaviors. Alternatively, firms can conduct experiments in which they compare the outcomes of similar individuals randomly exposed to different treatments.
Unfortunately, both methods have limitations. Although observational methods benefit from the large amount of data that are relatively cheap to obtain, they have been documented as severely biasing estimates of causal effects due to endogeneity (Lewis et al. (2011), Blake et al. (2015), Ursu (2018), Gordon et al. (2019)). While experiments can address this endogeneity concern, small-scale experiments often produce imprecise estimates. For example, Lewis and Rao (2015) show that advertising experiments require more than 10 million person-weeks to accurately measure the return to advertising. Large-scale experiments can be precise but too expensive to run because experiments can negatively affect firms' profits and user experiences. Thus, developing efficient methods to improve the use of observational and experimental data is particularly important for accurate causal effect measurement.
This paper highlights two important aspects of online experiments that can be used to improve estimation efficiency: 1) the random selection of experimental units and 2) the abundance of relevant first-stage covariates. First, an online experiment often randomly splits the website traffics into a large observational group and a small experimental group. This random splitting suggests that users in these two groups share similar characteristics. Second, many user characteristics are observed by the website and often used as inputs to an algorithm that generates customized marketing mix variables. This implies that firms have extensive knowledge over how the endogenous marketing mix variables in the observational group are generated. From an econometrician's perspective, this setting has many covariates that satisfy the first-stage relevance condition.
We propose a new method that combines observational and experimental data for causal effect estimation in settings like this. When compared to the estimator that only uses the experimental data, our method theoretically increases the estimation efficiency by up to 50% as measured by mean squared error (MSE). We also explore its implication for the selection of experimental units from an experimental design perspective. We show that the estimation efficiency can increase further if the experimental design incorporates the knowledge of how X is endogenously generated. We apply our method to measure the effect of online hotel rankings on clicks using datasets in Ursu (2018). We show that our method can increase efficiency by 25% under a common experimental design and by 40% under a novel experimental design.
The remainder of the paper is organized as follows: Section 2 gives a literature review. Section 3 presents our setup and key assumptions. Section 4 introduces the estimation method. Section 5 discusses the theoretical efficiency gain and its implications for experimenters and firms. Section 6 provides methods that are robust under model misspecifications in observational data. Section 7 uses simulation to illustrate the efficiency gain in different situations. Section 8 presents our results on the effect of hotel rankings on clicks using datasets in Ursu (2018). Concluding remarks are offered in Section 9.

Literature Review
We give a short overview of our methods before comparing them with existing methods. One of our methods first uses the observational data to derive a biased IV estimate of the causal effect and then correct this bias using the experimental data. We show that this bias-corrected estimator is uncorrelated with the benchmark estimator that only uses the experimental data, and can thus be combined with the benchmark estimator to improve efficiency.
Our estimation strategy is closely related to that of Imbens and Lancaster (1994), who combine macro and micro data to improve estimation efficiency. Our method is similar in that one dataset is sufficient for identifying the parameter of interest, while the moment conditions from the additional dataset help improve efficiency. However, Imbens and Lancaster (1994) consider a setting in which the underlying data in both datasets are drawn from the same distribution. Their method cannot be directly applied to our case because the variable of interest is endogenously determined in the observational data but randomized in the experimental data.
Our method relaxes the assumptions required to improve the estimation efficiency using pre-experimental data, which are a form of observational data. One popular method by Deng et al. (2013) incorporates pre-experimental variables as additional covariates into the experimental analysis. Their method requires individuals to be observed both before and during the experiment, and is conceptually different from our method that can be applied even if individuals are only observed once. These two methods also have different determinants for efficiency gains. The method of adding pre-experimental covariates is most effective when the pre-experimental data contain variables that are highly predictive of the experimental outcome Y . Our method is most effective when the pre-experimental data contain variables that are highly predictive of the endogenous variable X. These two methods are also not exclusive: they can be applied together if the data requirements and assumptions are met for both methods.
Our method is also related to several contemporary studies that examine combining experimental and observational data, although in different contexts and for different purposes. Athey et al. (2020) consider combining datasets to estimate long term treatment effect where the long-term outcome in the experimental data is missing. Kallus et al. (2018) consider a setting in which the experimental data is non-representative but the observational data is endogenous. These works focus on cases when one dataset alone is not sufficient for identification. In contrast, we focus on increasing the efficiency, where the experimental data alone is sufficient for identification but the accuracy is limited by the sample size. Rosenman et al. (2020) also study improving the accuracy of the experimental estimator by incorporating the observational data. They shrink the unbiased experimental estimator to an additional biased observational estimator to derive a combined estimator that is biased but has lower MSE. The efficiency gain is relatively small if the bias of the additional estimator is large and the number of strata is small. Our method can also be understood as shrinking the experimental estimator to an additional estimator, but the additional estimator is derived differently, using a new source of information in the observational data. We also identify an assumption that ensures our additional estimator to be unbiased, allowing us to strictly improve efficiency without sacrificing the unbiasedness. These two methods are complementary and can be used together, as they rely on different assumptions and have different determinants for efficiency gains.
Our model for observational data is related to the literature by Conley et al. (2010) and Kippersluis and Rietveld (2018) that study plausibly exogenous model in which some covariates are considered as "plausible IV" because they satisfy the relevance condition but slightly violate the exclusion restriction. Conley et al. (2010) assume that there is a prior on how much the first-stage covariate violates the exclusion restriction. Our method also depends on the availability of first-stage covariates that satisfy the relevant condition. However, we allow the exclusion restriction to be strongly violated, and we do not require a prior on the degree of violation, because this can be informed by the experimental data.

Set-up
Consider a linear causal model where Y i is the outcome, X i is the focal variable of interest, Z i is the observed covariate, and U i is the unobserved covariate. β 1 and β 2 are respectively the causal effect of X i and Z i on Y i . β 1 is the primary parameter of interest, and β 2 is a nuisance parameter that may or may not be 0. The main difference between the observational data and experimental data is how X i is generated. Let G i ∈ {E, O} be the indicator for the data a unit i is drawn from, where E denotes experimental data and O denotes observational data. We assume the first stage equations for X i to be: γ is a non-zero first stage coefficient for the covariate Z i in the observational data and V i is the corresponding unobserved error term. We assume X i in the observational data is correlated with the unobserved covariate U i so that the observational data has an endogeneity problem: We also assume Z i violates the exlcusion restriction so that it is not an instrumental variable. This violation can be caused by a direct effect on Y i or a correlation with the unobserved covariate: Compared to the observational data, the experimental data can identify β 1 because X i is randomized and thus independent of U i : To summarize our set-up, the observational data alone cannot identify β 1 because X i is endogenous and Z i is not an instrumental variable. The experimental data are sufficient to identify β 1 , but the accuracy is limited by the sample size. By leveraging the covariate Z i that satisfies the first-stage relevance condition in the observational data, our method combines these two datasets to improve the efficiency in terms of mean squared error (MSE).

Assumptions
For our method to improve efficiency, a key assumption is that the joint distributions of (Z i , U i ) are similar between experimental and observational datasets. One sufficient condition is that the covariance and variance are identical in the two distributions: This assumption can be rationalized if a unit i with attributes (Z i , U i ) is first sampled from the population and then randomly assigned into the experimental or observational group, independent of (Z i , U i ): Assumption 1 (Random Selection of Experimental Units) This assumption ensures that the distributions of (Z i , U i ) are identical between observational and experimental data. Figure 1 illustrates a common experimental procedure in the digital environment that satisfies this assumption: Figure 1: An experimental design example that randomly assigns users to the experimental/observational group Worth discussing are two aspects of our set-up. First, we assume linearity and additive separability in our model, which can be easily violated if the model is non-linear or there exists interaction between X i and U i . Second, we assume that the covariance between the two datasets are exactly the same, which can be violated if the experimental and observational populations are slightly different. These two assumptions are unnecessary if the goal is to improve the MSE, which is affected by both the bias and variance. When these assumptions are satisfied, our method strictly improves the accuracy by reducing the variance of the estimator without increasing the bias. When these two assumptions are slightly violated, our method can still reduce the overall MSE through a reduction in variance at the potential cost of slightly increasing bias. We provide a more detailed discussion in the context of model misspecification in Section 6 and provide an example of nonlinear model in Appendix B.

Intuition
To develop intuition, consider the scenario in which we incorrectly use Z i as an IV to estimate β 1 in observational data . We define the probability limit of this estimator as The estimator is biased because Z i violates the exclusion restriction, either due to a direct effect on Y i or an unobserved correlation with U i . Let b 2 denote the violation of such exclusion restriction: Lemma 1 The bias of the incorrect IV estimator is b 2 γ The proof of this lemma is given in the Appendix A.1. When using observational data alone, this bias typically cannot be corrected because the violation of exclusion restriction b 2 is unknown and cannot be estimated. However, approximating this violation of exclusion restriction becomes possible with experimental data: Lemma 2 Under Assumption 1, the violation of exclusion restriction b 2 can be identified by regressing Y i on Z i using experimental data.
The proof of this lemma is given in the Appendix A.2.
To summarize our intuition, the incorrect IV estimator in the observational data is more useful when its bias can be quantified, and the experimental data help quantify such bias. Next we propose three different estimation procedures that are qualitatively similar but give different intuitions: 1) weighting, 2) regularized regression, and 3) GMM. These methods are qualitatively the same, but give different intuitions to help understand the source of efficiency gain.
We prove Lemma 3 and Lemma 4 in Appendix A.3. We give a visual illustration of this procedure in Figure 2. where λ is a non-negative hyperparameter that determines how much to penalize the deviation from the constraint. Intuitively, λ depends on how much we believe the violation of such constraint can be attributed to the inaccuracy of ( β 1 , b 2 ).
When λ = 0, it implies that we ignore information from observational data, and the optimization problem is equivalent to the one using only experimental data. A small value of λ can be rationalized if the observational dataset is small, such that a violation of the constraint can be explained by the inaccuracy of ( b IV O , γ O ) rather than the inaccuracy of ( β 1 , b 2 ).
When λ = ∞, it implies that we heavily rely on the constraint. A large λ can be justified if the observational data is large such that the precision of ( Then any violation of the constraint must be attributed to the inaccuracy of ( β 1 , b 2 ). When λ = ∞, the problem then becomes a constraint optimization: Since we are adding this additional constraint into the original experiment-only optimization procedure, when this constraint is correct, our method can strictly improve efficiency.

GMM
We also consider a GMM approach that can easily calculate standard errors using standard procedures. Equation 1 is not suitable for direct GMM estimation however, because one of the parameters β 2 cannot be identified when Cov(U i , Z i ) = 0. To derive relevant moment conditions, it is convenient to work with a residual i that is determined by identifiable parameters (β 1 , b 2 ): such that i and Z i are not correlated in both the observational and experimental data: We prove Lemma 5 in Appendix A.4. Note i is different from U i and can be understood as a transformation of U i that removes the correlation with the observed covariate Z i .
Following Lemma 5, we can derive two moment conditions from the experimental data and one extra moment condition from the observational data: Denote the vector of moment functions as g: The GMM estimator can be written as: where W is a non-negative definite weighting matrix. The optimal W * depends on the covariance matrix of the moment conditions, which can be estimated by running the standard feasible GMM. We discuss the optimal weighting matrix when i satisfies homoscedasticitiy and non-autocorrelation in Section 5.

Method Comparison
These three methods are qualitatively similar but offers different intuitions. All of these methods involve the selection of a hyperparameter: the weight in the weighting method, the regularization parameter λ in the regularization regression method, and the weighting matrix W in the GMM method. In practice, if the researchers believe the model is correctly specified, we recommend researchers to use the GMM approach, because the weighting matrix and standard errors can be easily determined using the existing frameworks such as two-step feasible GMM. If the model may be moderately misspecified however, we recommend researchers to use a regularization approach that makes use of cross-validation for hyper-parameter tuning. We discuss this cross-validation procedure in detail in Section 6.

Efficiency Gain
In this section we illustrate the efficiency gain using the GMM approach. It is convenient to first formulate the estimation problem in matrix notation: and the unit is ordered by group g ∈ {E, O} such that Since the exact efficiency gain depends on the distribution of (X i , Z i , i ), we focus on the simple of case of homoscedasticity and nonautocorrelation 1 Assumption 2 (Homoscedasticity and Nonautocorrelation): This assumption is not essential but helps illustrate the determinants of efficiency gain.
Theorem 1 If Assumptions 1 and 2 are satisfied, an optimal weighting matrix W satisfies W = Ω −1 where, and the GMM estimator using this weighting matrix has an asymptotic variance of: In comparison, the asymptotic variance of the experiment-only estimator is: A comparison of these two variances suggests that the efficiency gain is determined by 1) the proportion of observational data, 2) the relevance of Z i as a first-stage covariate, and 3) the relative variance of X E and X O : The relative variance of the two estimators can be written as: and simplifies the asymptotic variance. The proof for this theorem is provided in Appendix A.5

Implications For Firms Conducting Experiments
To understand the magnitude of the efficiency gain, we make two additional auxiliary assumptions that are not uncommon within firms. These two assumptions are not essential to our method but help illustrate the order of magnitude of our efficiency gain.
This assumption can be rationalized if firms have perfect knowledge over how X i is endogenously generated, and Z i is aggregated from all covariates used to generate X i in the observational data such that X i = γZ i for all G i = O. With perfect knowledge, there are no unobservables that affect X i other than covariates represented by Z i . 2 This assumption is easily satisfied in ranking experiments in online settings, where positions of ads or hotels listed by a website are randomly shuffled. It is also satisfied if the experimental X is sampled from the marginal distribution of the observational X.
Lemma 7 If Assumptions 1 and (2-4) are satisfied, the efficiency gain is bounded by the proportion of the observational data Given the above assumptions, firms can reduce the variance of the estimator by up to 50%, which implies that half of the experimental data is required to achieve the same accuracy when observational data is incorporated.

Implications for Experimental Designs
. We can relax this assumption to consider the case when Assumption 1 is satisfied, but the sampling of experimental units is non-random and depends on Z i such that Lemma 8 If Assumption 1 and 2 are satisfied, the efficiency gain of the GMM method under an optimal weighting matrix satisfies: The proof of this lemma is provided in the Appendix A.6. To understand the magnitude of the efficiency gain in this general case, consider the case when proportion of observational data approaches 1: Lemma 9 If Assumptions 1-4 are satisfied, and π O → 1: The efficiency gain of incorporating the observational data therefore increases if the variation of Z i in the experimental data increases. Intuitively, this additional efficiency gain comes from a more accurate measurement of b 2 , the violation of the exclusion restriction. When V ar(Z i |G i = E) is large such that b 2 is precisely measured, the observational data can be better leveraged.
V ar(Z i |G i = E) is a moment determined by experimenters who can choose an assignment rule that increases the variation of Z i . After sampling a unit i with observed covariate Z i from the population, experimenters can randomly assign this unit into the experimental group based on a probability that depends on Z i . For example, if Z i is binary with p(Z i = 1) = p 1 , the experimenter can design a balanced sampling procedure to make sure P

Robustness to Model Misspecification
In this section we demonstrate how our method can be generalized to settings when the experimental data is still sufficient for identification, but the observational model is misspecified. Recall that we assume a model with additive separability that rules out any interactions in Equation 1. We also assume the observational units to be similar to the experimental units in Assumption 1 . We relax these requirements to allow the model to include interactions between X and (U, Z), and the observational sample to be slightly different from the experimental sample. These requirements are not essential for running a regression using only the experimental data to obtain an unbiased estimate of the average causal effect for the experimental population. However, they are important for ensuring unbiasedness after the incorporation of observational data. Our method is still valuable if the goal is to minimize mean-squared error (MSE), which does not only include bias but also the variance. To alleviate the concern of model misspecification, we introduce a cross-validation procedure that help the researchers determine: 1) whether they should only use the experimental data or incorporate the observational data using our combined method and 2) a tuning procedure of the hyperparameter that is robust to model misspecification.
The key of our cross-validation strategy is to cross validate on out-of-sample experimental data. Intuitively, a better causal estimator should more accurately predict Y given a randomized X. The experimenter can therefore test the usefulness of a combined estimator β 1 combine relative to the experiment-only estimator β 1 E using out-of-sample experimental data. It is possible that the experiment-only estimator is unbiased but the combined estimator has low MSE due to a reduction in variance. This validation strategy is different from a classical cross validation method that randomly splits the entire sample. In contrast, this strategy only split a subset of datasets in which we believe our model is correctly specified, leveraging the assumption that the experimental data alone is sufficient for causal identification,.
One method is to use a leave-one-out cross-validation (LOOCV) strategy, where the leave-one-out cross-validation error is defined as: ) are estimated by omitting the ith observation in the experimental data. We can calculate the LOOCV error for the experiment-only estimator β 1 E,(−i) and combined GMM estimator β 1 combine, (−i) to compare their average performance. This helps us determine if incorporating observational data can reduce the MSE despite possible model misspecification.
Since this cross validation strategy can be used to compare estimators, it can also help tune hyperparameters associated with a class of estimators. Next we demonstrate how to improve the efficiency under misspecification of observational model using 1) the weighting method and 2) the regularization method.

Weighting Method with Cross Validation
Recall that our weighting method combines two estimators β 1 E and β 1 O . β 1 E is an estimator that only depends on the experimental data. When the experimental model is correctly specified, β 1 E is an unbiased estimator whose accuracy may be limited by the size of experimental data. In comparison, β 1 O is an estimator that partly relies on the observational data. When the observational model is misspecified, β 1 O can be a biased estimator with small variance. w E and w O are the hyperparameters that require tuning. Researchers can pick (w E , w O ) that has the lowest LOOCV error.
This approach makes use of the ensemble intuition that combines multiple estimators or models, where the weights are tuned using cross validation. The key difference is that we are validating using experimental data only.

Regularized Regression with Cross Validation
Recall that our objective function in the regularized regression approach is: are estimated using observational data and λ is a hyperparameter that can be tuned. Intuitively, λ should be large if the violation of this constraint can be mostly attributed to the inaccuracy of ( β 1 , γ O ). Therefore λ should be large if the observational data is large and the model is correctly specified. If the model is moderately misspecified such that the term is non-zero but small, then λ should be smaller but still non-zero because it can reduce the variances.
This method is similar to other shrinkage methods in which the shrinkage hyperparameter is tuned using cross validation. The key difference is that instead of shrinking parameters to 0, as in Lasso and Ridge Regression, we shrink toward moments implied by the model assumptions.

Simulation
In this section we use simulation to illustrate the effectiveness of our method. We start with a linear causal model where X i is endogenously determined by (Z i , V i ) in the observational group but randomly assigned in the experimental group: is drawn from a distribution unknown to researchers: V i is the residual term in the observational first stage and therefore by definition uncorrelated with Z i . We set . In addition, we assume a unit is randomly selected into the experimental group with probability π E : We draw 10000 samples from the above distribution, where each sample contains n E = 100 experimental units. Within each sample s, we calculate the experiment-only estimator β 1 E,s and the combined GMM estimator β 1 Combine,s and then summarizes the bias, variance and MSE. Figure 3 compares the distribution of these two estimators when γ = 0.95 and π O = 0.95. It shows that the combined GMM estimator is unbiased and on average more accurate.  For what follows, we fix γ = 0.95 and π O = 0.95 such that the relevance of Z and the proportion of observational data are both high. We then compare the performance of our methods with other alternatives when the model is correctly and incorrectly specified. Table 1 compares our combined GMM estimator with OLS and IV estimator that either use only experimental or observational data.

Comparison with Other Methods
In the last column we calculate the MSE of these estimators relative to that of the experiment-only estimator. Our combined GMM estimator reduces the MSE by almost 50%. Methods based on observational data have high MSE due to high bias: OLS is biased because X is highly endogenous, and IV is biased because Z is not an instrumental variable as it violates the exclusion restriction .

Estimation with Improved Experimental Design
We demonstrate how to design the experiment differently to further increase the efficiency gain of incorporating observational data based on Lemma 9. Instead of randomly assigning a unit into the experimental or observational group, experimenters can select an experimental group that has large variation in the experimental Z i to more accurately estimate the violation of the exclusion restriction b 2 . One method is to order the sampled units by Z i , and assign a unit into the experimental group with non-zero probability only if the unit is in the top or bottom Q quantiles. 3 Let {q i } N i=1 denote the quantiles of units ordered by Z i , the assignment rule is: is an assignment probability that ensures π E proportion of units are assigned into the experimental group. Figure 5 shows the relative MSE when we hold π E = 0.05 and sample experimental units from the top/bottom Q quantiles for different values of Q. When Q = 0.5, the sampling procedure is equivalent to random sampling, and the efficiency gain is close to 50% as predicted by Lemma 7. When Q = π E 2 , the sampling procedure is equivalent to assigning all the top and bottom π E 2 quantiles of units into the experimental groups and maximizes V ar(Z i |G i = E). The relative MSE is around 20%, indicating that only 20% of the experimental data is needed to achieve the same level of accuracy. When the distribution of (Z i , U i , V i ) is normal, this non-random sampling procedure does not violate Assumption 1, ensuring our estimators to be unbiased. This unbiasedness no longer holds if Assumption 1 is violated under a different distribution. However, if this assumption is only moderately violated, our method can still improve the efficiency through the reduction in variance at the cost of slightly increasing the bias.

Robustness to Interactions
Next we examine whether our method is still effective when the model is misspecified due to interactions between X and U . We assume that the researcher believes that the causal model follows Equation 7 but the actual causal model has an additional interaction term: Figure 6 shows the relative MSE under different values of θ. Overall the efficiency gain is still large, with a relative MSE of around 60% when |θ| = 2. This is because the combined estimator happens to remain unbiased when (Z i , U i , V i ) are normally distributed. Under this normality, incorporating observational data can still strictly improve the MSE by reducing the variance without increasing the bias. The efficiency gain decreases slightly as |θ| increases, because homoscedasticity in Assumption 2 no longer holds.

Robustness to Misspecified Covariance
Another form of model misspecification is that the distributions of (Z i , U i , V i ) are actually different between the observational and experimental samples but the researchers believe the distributions are identical. For example, the observational sample includes all users who arrive at the website in the past, and the experimental sample represents all users who arrive this week. When the two datasets are sampled at different points in time, they may have different distributions. The covariance Cov(Z i , U i |G i = g) may therefore depends on g ∈ {E, O}: which violates Assumption 1 when ρ O zu = ρ E zu . We fix ρ E zu at 0.4 and examine how different values of ρ O zu affect the efficiency gain. Figure 7 shows the relative MSE under different values of (ρ The combined GMM approach is more efficient than the experiment-only estimator if the difference in correlation is smaller than 0.2 but is significantly worse when the difference in correlation continues to increase. We also test the performance of the robust weighting method that cross validates on experimental data. Although this cross-validation approach only is not as effective as the combined GMM approach when the model is correctly specified, it is more robust to model misspecification. The cross-validation method still has non-negative efficiency gain when the correlation difference is around 0.3. Furthermore, when the correlation difference is as large as 0.4, the cross validation approach only performs slightly worse than the experiment-only estimator. This simulation suggests that researchers can still use the combined GMM approach when the experimental and observational data are slightly different. However, if researchers believe that the correlation between the observed attribute Z i and unobserved attribute U i are very different across the datasets, researchers should either discard the observational data or use the robust weighting approach.

Empirical Application: Online Hotel Ranking
We apply our method to measure the effect of website positions on clicks using the Expedia ranking dataset in Ursu (2018). The dataset consists of consumers' search queries for hotels on Expedia, with each search query containing a ranked list of hotels. For each hotel in a query, we observe its rank, characteristics, and click outcome, and there are 4.5 million such observations. Two thirds of the data are observational in which the hotel rankings are ordered by relevance, while one third of the data are experimental in which the hotel rankings are randomized.
We assume a linear causal model: where each observation corresponds to a queried-hotel i. The position is the numerical rank of the hotel in that query, and we assume it to be correlated with other hotel characteristics in the observational data: One challenge is to derive a relevant first-stage covariate Z i to predict positions that may depend on locations and other characteristics. We select locations that have more than 10, 000 observations and for each location apply a random forest procedure that uses the hotel characteristics to predict the hotel position in the observational data to estimate f : The experimental and observational datasets respectively contain N E = 452, 974 and N O = 833, 798 observations in these selected locations. Table 2 shows the observational first-stage in which Z is a relevant covariate that explains around 57% of variations in terms of R 2 . Note: * p<0.1; * * p<0.05; * * * p<0.01

Efficiency Gain Using a Random Experimental Sample
We examine the effectiveness of our methods when units are randomly assigned into the experimental and observational groups. We focus on a case when the observational group is large with n O = N O 2 units and the experimental group is small with n E = N E 100 units. This sample can be simulated by randomly drawing n O units from the observational data and n E units from the experimental data. We repeatedly draw M = 10, 000 such samples and perform estimation on each sample. In order to evaluate the performance of different estimators, we approximate the true causal effect β 1 by estimating it using the original experimental dataset in these selected locations. This approximation is sufficient for our purpose because the size of the original experimental dataset is 100 times the size of the experimental sample. Table 3 reports the bias, variance and MSE of different estimators discussed in Section 7.1 across 10,000 replications. To characterize the efficiency gain, we compare the MSE of these methods with a benchmark that uses only a random sample of experimental data for estimation. The OLS using observational data has large bias due to endogeneity. The incorrect IV approach also has large bias, because the instrumental variable Z i violates the exclusion restriction. In comparison, our combined GMM has a efficiency gain of 24.2% when using a first-stage covariate that explains 57% of variation in ranking. If firms have more information on how X in generated in the observational data, this efficiency can further increase. Notes: Bias 2 , Variance, and MSE are all multiplied by 10 6 for readability.

Efficiency Gain Under Improved Experimental Design
In the previous section we study an experimental procedure that randomly assigns units into the experimental group. In this section we examine alternative experimental designs that adjust the sampling procedure of experimental units. This adjustment can be achieved by assigning a unit to the experimental group with a probability that depends on Z i as discussed in section 7.2. The experimental group can be simulated by ordering the experimental population by Z i , and drawing n E units from the top and bottom Q quantiles. The observational group can still be approximated by randomly drawing n O units from the observational data because n O is large relative to n E . To illustrate this alternative experimental procedure, Figure 8 shows the distribution of 1) all experimental Z i , 2) Z i randomly sampled from the original experimental data, and 3) Z i sampled from the top/bottom 20% quantiles of the experimental data.  Figure 9 summarizes the distribution of estimates under the two experimental designs. The first figure summarizes a benchmark in which a random experimental sample is used for the estimation. The second figure summarizes the GMM estimators that combine the random experimental sample with a large observational sample. The third figure summarizes GMM estimators that combine a large observational sample with experimental units sampled from the top and bottom 20% quantiles ordered by Z i . A comparison of the first and second figures shows that incorporating observational data into the experimental analysis can improve the efficiency. A comparison of the second and third figure shows that researchers should not only consider incorporating the observational data into the experimental analysis but also into the experimental design, which has even larger efficiency gain.  Table 4 reports the bias, variance, and MSE of estimators when we adjust the experimental design to sample units from the top and bottom Q quantiles ordered by Z i for different values of Q. All of these alternative experimental designs have lower MSE than the design that randomly assigns units into the experimental group, but the bias and variance differ. Under a moderate adjustment in which an experimental unit is sampled the top/bottom 30% or 20% quantiles, the efficiency gain is higher than 35% and the bias is negligible relative to the variance. The bias becomes larger as we move to more extreme experimental designs that sample experimental units from the top/bottom 10% or 5% quantiles. The efficiency gain varies from 28% to 42%. This exercise suggests that it is valuable to design an sampling procedure that increases V ar(Z i |G i = E) for a relevant first-stage covariate Z i . A moderate adjustment of the experimental design can improve the M SE by further reducing the variance at the cost of only slightly increasing the bias. But researchers should be cautious with a more extreme adjustment of the experimental design, which may lead to large bias increase that cannot be offset by the variance reduction.

Limitations of the Expedia Application
This application has several limitations. As discussed in Ursu (2018), the dataset has some selection bias and therefore the distribution of queried hotels in the experimental data set is different from that in the observational dataset. Ursu (2018) developed a structural model that help address this concern of selection bias. In addition, our reduced form model may be misspecified in several ways. First, the position effect can be nonlinear since the effect of changing from the second place to the first may be different from changing from the third place to the second. Second, we assume a linear probability model of click. Third, we assume that each observation is i.i.d but the positions and clicks may be interdependent across hotels in the same query. Alleviating these concerns requires developing a rich structural model of consumer search and estimating it using a full information maximum likelihood approach. Since dealing with model misspecification and developing structural model is not the primary goal of this paper, we leave this to future extension. Despite these limitations, this dataset has several advantages for evaluating our method's effectiveness. First, the dataset contains both observational and experimental data, eliminating the need for simulation. Second, it contains covariates that satisfy our first-stage relevance condition. Variables, such as rating and price, are likely to affect hotel rankings. Third, the dataset is relatively large, allowing us to approximate the true causal parameter using the entire set of experimental data. An accurate approximation of the underlying truth allows us to evaluate parameters estimated using only a small subsample of the experimental data. We therefore believe this is still a useful empirical dataset for demonstrating the usefulness of our method.

Conclusion
In this paper, we develop a new method that combines experimental and observational data to improve estimation efficiency. We focus on a setting in which the experimental data are sufficient for identification but limited in size, and the observational data are large but suffer from endogeneity and a lack of instrumental variables. We identify a common experimental design in which units with attributes (Z, U ) are randomly selected into the experimental group, and show that our method can reduce the variance by up to 50% without increasing the bias. The efficiency gain is affected by the size of the observational data and the relevance of the first-stage covariate Z. Intuitively, an incorrect IV estimate obtained using the observational data is more useful when its bias can be quantified and corrected. The experimental data can help quantify such bias.
We also demonstrate alternative experimental designs that can further reduce the variance at the potential cost of slightly increasing the bias. This is achieved by assigning a unit i into the experimental group with a probability that depends on Z i . Intuitively, this alternative experimental design can more accurately estimate the bias of the incorrect IV estimate, allowing us to better leverage the observational data. We apply our method to the Expedia experiment and show that our method can reduce the variance by about 25% when experimental units are randomly selected and by 40% when the selection of experimental units depend on the first-stage covariate Z.
Our method is not without limitations. It is most useful when the model is linear and the joint distributions of the observed and unobserved attributes (Z i , U i ) are similar between the experimental and observational samples. When these assumptions are satisfied, our method can strictly reduce the variance without increasing the bias. When the assumptions are moderately violated, our method can still improve the overall MSE at the cost of slightly increasing the bias. However, when these assumptions are strongly violated, it is possible our method can lead to a larger bias that cannot be compensated by the reduction of variance. We develop a cross-validation approach that is more robust to model misspecification and recommend researchers to use this robust method when they believe that the underlying samples are very different.

A Proof
A.1 Proof of Lemma 1 Lemma 1: The bias of the incorrect IV estimator is b 2 A.2 Proof of Lemma 2 Lemma 2: Under Assumption 1, the violation of exclusion restriction b 2 can be identified by regressing Y i on Z i using experimental data. Proof : A.3 Proof of Lemma 3 and 4 ) are estimated using experimental data, these pairs of estimates are independent: Because X i and Z i in the experimental data are independent and used to estimate Plugging Equations 9 and 10 into the covariance: Lemma 4: Under Lemma 3 the weighting is optimal if each estimate is weighted in inverse proportion to its variance: Taking the first order condition subject to the constraint that w O + w E = 1 we get:

A.4 Proof of Lemma 5
Recall that b 2 is defined in equation 4 as In the observational data: Given Assumption 1, so the same relationship holds for the experimental data: Cov( i , Z i |G i = E) = 0.

A.5 Proof of Theorem 1
We provide a general proof for the multivariate case and later simplifies the results to the univariate case. Recall that the moment function is with π E and π O proportion of units in the experimental and observational groups.
The inverse of the covariance matrix of the moment function Ω is: which can be simplified under Assumption 2: with block inversion, the variance of β 1

GM M
can be written as When both X i and Z i are univariate random variables with 0 means, the variance can be simplified to If Assumption 1 is satisfied such that V ar(Z i |G i = E) = V ar(Z i |G i = O) = V ar(Z i ). The variance can be further simplified to This completes the proof of Theorem 1.

A.6 Proof of Lemma 8
The proof of theorem 1 shows that: The relative variance can be written as: This completes the proof for Lemma 8.
can be understood as the additional term that characterizes the relative variance of Z between observational data and experimental data because it can be written as:

B Extension to Nonlinear Model
Most of the previous sections focused on linear model. It is well known that the endogeneity in nonlinear models is difficult to handle and has only been extensively studied in a few special cases. In this section we illustrate how the observational data can be incorporated into the case of binary probit with endogenous regressors following a textbook example in Wooldridge (2010, Section 15.7.2), where: We hold the first stage-equation to be the same as in Equation 2: The standard probit assumes that (U i , V i ) follows a standard normal distribution that is independent of the instrumental variable Z i . To make the case more general, we allow Z i to depend on U i 4 : v Z i is therefore not an instrumental variable because it either directly affects Y * i , or correlated with unobservables U i .
As in our linear case, not all parameters can be separately identified in this set-up and we need to aggregate some parameters for estimation convenience. We define b 2 as b 2 ≡ β 2 + ρ zu which captures both the direct effect of Z on Y * and an indirect correlation with the unobservables U . Let Θ denote the set of unknown parameters: Θ = {β 1 , b 2 , ρ uv , γ, σ v } A maximum likelihood estimator that only uses the experimental data chooses (β 1 , b 2 ) to maximizes: The log-likelihood of the observational data can be written as The data can be combined by optimizing the sum of these two log-likelihoods: To understand why incorporating the observational data can help improve efficiency in this probit case, consider when the size of the observational data is infinite. (γ, σ v ) are identified using the first-stage equation, where Following Wooldridge, under normality, the second stage can be written as: A probit of Y i on (X i , Z i ) can then consistently identify two constraints: Since (γ, σ v ) are already identified in the first-stage, (b 2 , β 1 , ρ uv ) are the three unknowns left. If b 2 = 0, the problem is reduced to an endogenous binary probit with Z i being an instrumental variable. Then (β 1 , ρ uv ) are just identified under C 1 and C 2 . However, because b 2 is unknown, the observational data alone are not sufficient for identification. Although the observational data are not sufficient for identification, it can help improve efficiency of the experimental data by imposing additional constraints onto the maximum likelihood problem: Intuitively, when the two constraints C 1 and C 2 are precisely measured, any deviations from these constraints must be attributed to the inaccuracy of (β 1 , b 2 ). Incorporating the observational data can therefore help improving the efficiency.