Experimental Design in Two-Sided Platforms: An Analysis of Bias
We develop an analytical framework to study experimental design in two-sided marketplaces. Many of these experiments exhibit interference, where an intervention applied to one market participant influences the behavior of another participant. This interference leads to biased estimates of the treatment effect of the intervention. We develop a stochastic market model and associated mean field limit to capture dynamics in such experiments, and use our model to investigate how the performance of different designs and estimators is affected by marketplace interference effects. Platforms typically use two common experimental designs: demand-side ("customer") randomization (CR) and supply-side ("listing") randomization (LR), along with their associated estimators. We show that good experimental design depends on market balance: in highly demand-constrained markets, CR is unbiased, while LR is biased; conversely, in highly supply-constrained markets, LR is unbiased, while CR is biased. We also introduce and study a novel experimental design based on two-sided randomization (TSR) where both customers and listings are randomized to treatment and control. We show that appropriate choices of TSR designs can be unbiased in both extremes of market balance, while yielding relatively low bias in intermediate regimes of market balance.
\subsection{Estimation with the $\ensuremath{\mathsf{TSR}}$ design}
\label{ssec:TSR_est}

The preceding sections reveal that each of the naive $\ensuremath{\mathsf{LR}}$ and $\ensuremath{\mathsf{CR}}$ estimators has its virtues, depending on market balance conditions.  In this section, we explore whether we can develop $\ensuremath{\mathsf{TSR}}$ designs and estimators in which $a_C$ and $a_L$ are chosen {\em as a function of} $\lambda/\tau$, to obtain the beneficial asymptotic performance of the naive $\ensuremath{\mathsf{CR}}$ estimator in the highly-demand constrained regime, as well as the $\ensuremath{\mathsf{LR}}$ estimator in the highly supply-constrained regime. We also expect that an appropriate interpolation should yield a bias for $\ensuremath{\mathsf{TSR}}$
that is comparable to, if not lower than, $\ensuremath{\mathsf{CR}}$ and $\ensuremath{\mathsf{LR}}$ in intermediate regimes of market balance.

Recall the naive $\ensuremath{\mathsf{TSRN}}$ estimator defined in \eqref{eq:naive_TSR_est}, and in particular the steady-state version of this estimator.  Suppose the platform observes $\lambda/\tau$; note that this is reasonable from a practical standpoint as this is a measure of market imbalance involving only the overall arrival rate of customers and the average rate at which listings become available.  For example, consider the following heuristic choices of $a_C$ and $a_L$ for the $\ensuremath{\mathsf{TSR}}$ design, for some fixed values of $\b{a}_C$ and $\b{a}_L$:
\begin{equation}
a_C(\lambda/\tau) = \left(1 - e^{-\lambda/\tau}\right) + \b{a}_C e^{-\lambda/\tau}; \quad a_L(\lambda/\tau) = \b{a}_L\left(1 - e^{-\lambda/\tau}\right) + e^{-\lambda/\tau}. \label{eq:aC_aL_naive_TSR}
\end{equation}
Then as $\lambda/\tau \to 0$, we have $a_C(\lambda/\tau) \to \b{a}_C$ and $a_L(\lambda/\tau) \to 1$, while as $\lambda/\tau \to \infty$ we have $a_C(\lambda/\tau) \to 1$ and $a_L(\lambda/\tau) \to \b{a}_L$.\footnote{Our choice of exponent here is somewhat arbitrary; the same analysis follows even if we replace $e^{-\lambda/\tau}$ with $e^{-c\lambda/\tau}$ for any value of $c > 0$.}. With these choices, it follows that in the highly demand-constrained limit ($\lambda/\tau \to 0$), the $\ensuremath{\mathsf{TSRN}}$ estimator becomes equivalent to the naive $\ensuremath{\mathsf{CR}}$ estimator, while in the highly supply-constrained limit ($\lambda/\tau \to \infty$), the $\ensuremath{\mathsf{TSRN}}$ estimator becomes equivalent to the naive $\ensuremath{\mathsf{LR}}$ estimator.  In particular, using Propositions \ref{prop:Qij_demand_constrained} and \ref{prop:Qij_supp_constrained}, it is straightforward to show that the steady-state naive $\ensuremath{\mathsf{TSRN}}$ estimator is unbiased in {\em both} limits; we state this as the following theorem, and omit the proof.%
\begin{corollary}
\label{cor:naive_TSR}
For each $\lambda/\tau$, consider the $\ensuremath{\mathsf{TSR}}$ design with $a_C$ and $a_L$ defined as in \eqref{eq:aC_aL_naive_TSR}.  Consider a sequence of systems where either $\lambda/\tau \to 0$, or $\lambda/\tau \to \infty$.  Then in either limit:
\[ \widehat{\ensuremath{\mathsf{GTE}}}^{\ensuremath{\mathsf{TSRN}}}(\infty | a_C(\lambda/\tau), a_L(\lambda/\tau)) - \ensuremath{\mathsf{GTE}}\to 0. \]
\end{corollary}


We are also led to ask whether we can improve upon the naive $\ensuremath{\mathsf{TSRN}}$ estimator when the market is moderately balanced.  %
Note that the $\ensuremath{\mathsf{TSRN}}$ estimator does not explicitly correct for {\em either} the fact that there is interference across listings, or the fact that there is interference across customers. We now suggest a heuristic for correcting these effects, which we use to define two improved interpolating $\ensuremath{\mathsf{TSR}}$ estimators; these estimators are the fourth and fifth estimators appearing in Figure \ref{fig:numerics_homogeneous}, which we call ``$\ensuremath{\mathsf{TSR}}$-Improved (1)'' and ``$\ensuremath{\mathsf{TSR}}$-Improved (2)". These effects are visualized in Figure \ref{fig:tsr_comp_terms}.

First, abusing notation, let $\widehat{\ensuremath{\mathsf{GTE}}}^{\ensuremath{\mathsf{CR}}}(T|a_C, a_L)$ denote the estimator in \eqref{eq:naive_CR} using the same terms from a $\ensuremath{\mathsf{TSR}}$ design, and  dividing through by $a_L$ on both terms as normalization.  Similarly abusing notation,  let $\widehat{\ensuremath{\mathsf{GTE}}}^{\ensuremath{\mathsf{LR}}}(T|a_C, a_L)$ denote the estimator in \eqref{eq:naive_LR} using the same terms from a $\ensuremath{\mathsf{TSR}}$ design, and dividing through by $a_C$ on both terms as normalization.  Motivated by these naive estimators, we explicitly consider an interpolation between the $\ensuremath{\mathsf{LR}}$ and $\ensuremath{\mathsf{CR}}$ estimators of the form:
\begin{align}
\label{eq:interpolatingTSR}
 \beta \widehat{\ensuremath{\mathsf{GTE}}}^{\ensuremath{\mathsf{CR}}}(T|a_C,a_L)+(1-\beta)\widehat{\ensuremath{\mathsf{GTE}}}^{\ensuremath{\mathsf{LR}}}(T|a_C,a_L)& \\ = \beta \left(\frac{Q_{11}(T | a_C,a_L)}{a_Ca_L} - \frac{Q_{01}(T | a_C,a_L)}{(1-a_C)a_L}\right) &+ (1-\beta) \left(\frac{Q_{11}(T | a_C,a_L)}{a_Ca_L} - \frac{Q_{10}(T | a_C,a_L)}{a_C(1-a_L)}\right). \notag
 \end{align}
 

Now, consider the quantity $Q_{00}(T|a_C,a_L)/((1-a_C)(1-a_L)) - Q_{10}(T|a_C,a_L)/((1-a_C)a_L)$ in a $\ensuremath{\mathsf{TSR}}$ design. This is the (appropriately normalized) difference between the rate at which control customers book control listings, and the rate at which treatment customers book control listings.   Note that both treatment and control customers have the same utility for control listings, due to the $\ensuremath{\mathsf{TSR}}$ design, but potentially different utilities for treatment listings. Hence, the difference in steady-state rates of booking among control and treatment customers on {\em control listings} must be driven by the fact that treatment customers substitute bookings from control listings to treatment listings (or vice versa). This difference captures the ``cannibalization" effect (i.e., interference) that was found in $\ensuremath{\mathsf{LR}}$ designs in the  demand-constrained regime.

\begin{figure} 
    \centering
    \includegraphics[scale=0.35]{figs_other/tsr_competition_terms.png}
    \caption{Illustration of $\ensuremath{\mathsf{TSR}}$ design and competition effects. Intervention only applies when treatment customers view treatment listings (green cell). Suppose that the intervention makes the listing more attractive. %
    }
    \label{fig:tsr_comp_terms}
\end{figure}




Thus motivated, we can think of this difference as a ``correction term'' for the $\ensuremath{\mathsf{LR}}$ design from our interpolating $\ensuremath{\mathsf{TSR}}$ estimator in \eqref{eq:interpolatingTSR}. %
Using a symmetric argument we can also consider an appropriately weighted correction term associated to interference across customers in a $\ensuremath{\mathsf{CR}}$ design:  $Q_{00}(T|a_C,a_L)/((1-a_L)(1-a_C))-Q_{01}(T|a_C,a_L)/((1-a_C)a_L)$.  (Similar estimates were also studied in \cite{bajari2019double}; see the related work for further details on this work.)  See Figure \ref{fig:tsr_comp_terms} for an illustration of these competition effect estimates. %

We can weight these correction terms with different factors $k > 0$ to control their impact.  In addition, we can choose these weights in a market-balance-dependent fashion, based on the direction of market balance in which we have seen that the respective interference grows.  Combining these insights, for $\beta \in (0,1)$ and $k > 0$ we define a class of improved $\ensuremath{\mathsf{TSR}}$ estimators given by:
\begin{multline}
\label{eq:TSRI-A}
\widehat{\ensuremath{\mathsf{GTE}}}^{\ensuremath{\mathsf{TSRI\text{-}k}}}(T|a_C, a_L) = \\
\beta \left[\frac{Q_{11}(T | a_C,a_L)}{a_Ca_L} - \frac{Q_{01}(T | a_C,a_L)}{(1-a_C)a_L}-k(1-\beta)\left(\frac{Q_{00}(T|a_C, a_L)}{(1-a_C)(1-a_L)}-\frac{Q_{01}(T|a_C, a_L)}{(1-a_C)a_L}\right) \right] \\
+(1- \beta) \left[ \frac{Q_{11}(T | a_C,a_L)}{a_Ca_L} - \frac{Q_{10}(T | a_C,a_L)}{a_C(1-a_L)}-k\beta\left(\frac{Q_{00}(T|a_C,a_L)}{(1-a_C)(1-a_L)}-\frac{Q_{10}(T|a_C,a_L)}{a_C(1-a_L)}\right) \right] \ .
\end{multline}
Given market balance $\lambda/\tau$, we set $\beta=e^{-\lambda/\tau}$, and we choose $a_C$ and $a_L$ as in \eqref{eq:aC_aL_naive_TSR}.

In the limit where $\lambda/\tau \to 0$, note that $\widehat{\ensuremath{\mathsf{GTE}}}^{\ensuremath{\mathsf{TSRI}}-k} (T|a_C(\lambda/\tau),a_L(\lambda/\tau))$ approaches $\widehat{\ensuremath{\mathsf{GTE}}}^{\ensuremath{\mathsf{CR}}}(T|\b{a}_C)$ as expected.  Similarly, in the limit where $\lambda/\tau \to \infty$, $\widehat{\ensuremath{\mathsf{GTE}}}^{\ensuremath{\mathsf{TSRI\text{-}k}}} (T|a_C(\lambda/\tau),a_L(\lambda/\tau))$ approaches $\widehat{\ensuremath{\mathsf{GTE}}}^{\ensuremath{\mathsf{LR}}}(T|\b{a}_L)$.  It is straightforward to show that  $\widehat{\ensuremath{\mathsf{GTE}}}^{\ensuremath{\mathsf{TSRI\text{-}k}}}$ for any $k$ is  unbiased  in {\em both} the highly demand-constrained and highly supply-constrained regimes, since the correction terms play no role in the limits.

For moderate values of market balance,  both the cannibalization correction terms kick in, which lead to improvements over naive $\ensuremath{\mathsf{TSRN}}$ as seen in Figure \ref{fig:numerics_homogeneous}. 
To simplify the exposition, we only consider two factors $k=1,2$; we see that $\ensuremath{\mathsf{TSRI\text{-}1}}$ has lower bias than the naive $\ensuremath{\mathsf{TSR}}$, but $\ensuremath{\mathsf{TSRI\text{-}2}}$, which has a higher weight in front of the correction terms, has a lower bias than both naive $\ensuremath{\mathsf{TSRN}}$ and $\ensuremath{\mathsf{TSRI\text{-}1}}$, as well as the naive $\ensuremath{\mathsf{CR}}$ and $\ensuremath{\mathsf{LR}}$ estimators.  In Appendix \ref{app:simulations}, we explore the robustness of our results to other model primitives, specifically scenarios with smaller or larger utilities and the introduction of heterogeneity on one or both sides of the market. We find that the bias of $\ensuremath{\mathsf{CR}}$ and $\ensuremath{\mathsf{LR}}$ estimators can increase with the introduction of these factors, but remarkably the bias of the $\ensuremath{\mathsf{TSRI}}$ estimators remains low across the ranges that we study.  We emphasize the fact that the three $\ensuremath{\mathsf{TSR}}$ estimators presented here are examples to illustrate the potential for bias reduction using this new design. There is of course a much broader range of both $\ensuremath{\mathsf{TSR}}$ designs and estimators; some of these may offer even better performance.%


We conclude this section with two additional observations.  First, note that all our analysis in this section has been carried out in the mean field steady state; in particular, Figure \ref{fig:numerics_homogeneous} shows the bias of the estimators in steady state. 
For practical implementation, it is also important to consider the relative bias in the candidate estimators in the transient system, since experiments are typically run for relatively short time horizons. For discussion of the transient behavior for a finite time horizon, see Appendix \ref{app:transient}.  Second, in the next section, we discuss variance of the estimators we have studied.  There we find that the $\ensuremath{\mathsf{TSR}}$ estimators with the lowest bias also have the highest variance; in other words, there is a {\em bias-variance tradeoff}.



\begin{figure}
    \centering
    \includegraphics[width=0.5\textwidth]{figs_mean_field/homogeneous_listings_customers_mf_bias_vary_lambda_mod.png}
    \caption{Difference between estimator and $\ensuremath{\mathsf{GTE}}$ in steady state.  We consider variation in $\lambda/\tau$ by fixing $\tau = 1$ and varying $\lambda$; analogous results are obtained if $\lambda$ is fixed and $\tau$ is varied. We consider a market with homogeneous customers and homogeneous listings, pre-treatment. We set $\epsilon=1$ and $\alpha=0.5$. In the $\ensuremath{\mathsf{CR}}$ design, $a_C=0.5$. In the $\ensuremath{\mathsf{LR}}$ design, $a_L=0.5$. 
    Customers have utility $v=0.315$ for control listings and $\tilde{v}=0.394$ for treatment listings, which corresponds to a steady state booking probability of 20 percent in global control and 23 percent global treatment when $\lambda=\tau$. }
    \label{fig:numerics_homogeneous}
\end{figure}




\subsection{Estimation with the $\ensuremath{\mathsf{TSR}}$ design}

The preceding section reveals that each of the naive $\ensuremath{\mathsf{LR}}$ and $\ensuremath{\mathsf{CR}}$ estimators has its virtue, depending on market balance conditions.  In this section, we explore whether we can develop $\ensuremath{\mathsf{TSR}}$ designs and estimators in which $a_C$ and $a_L$ are chosen {\em as a function of} $\lambda/\tau$, to obtain the beneficial asymptotic performance of the naive $\ensuremath{\mathsf{CR}}$ estimator in the highly-demand constrained regime, as well as the $\ensuremath{\mathsf{LR}}$ estimator in the highly supply-constrained regime. \newgw{We also explore the performance of $\ensuremath{\mathsf{TSR}}$ in intermediate regimes of market balance.}

Recall the naive $\ensuremath{\mathsf{TSR}}$ estimator defined in \eqref{eq:naive_TSR_est}, and in particular the steady-state version of this estimator.  Suppose the \newgw{platform} \delgw{market} observes $\lambda/\tau$; note that this is reasonable from a practical standpoint as this is a measure of market imbalance involving only the overall arrival rate of customers and the average rate at which listings become available.  For example, consider the following heuristic choices of $a_C$ and $a_L$ for the $\ensuremath{\mathsf{TSR}}$ design, for some fixed values of $\b{a}_C$ and $\b{a}_L$:
\begin{equation}
a_C(\lambda/\tau) = \left(1 - e^{-\lambda/\tau}\right) + \b{a}_C e^{-\lambda/\tau}; \quad a_L(\lambda/\tau) = \b{a}_L\left(1 - e^{-\lambda/\tau}\right) + e^{-\lambda/\tau}. \label{eq:aC_aL_naive_TSR}
\end{equation}
Then as $\lambda/\tau \to 0$, we have $a_C(\lambda/\tau) \to \b{a}_C$ and $a_L(\lambda/\tau) \to 1$, while as $\lambda/\tau \to \infty$ we have $a_C(\lambda/\tau) \to 1$ and $a_L(\lambda/\tau) \to \b{a}_L$.\footnote{Our choice of exponent here is somewhat arbitrary; the same analysis follows even if we replace $e^{-\lambda/\tau}$ with $e^{-c\lambda/\tau}$ for any value of $c > 0$.}. With these choices, it follows that in the highly demand-constrained limit ($\lambda/\tau \to 0$), the naive $\ensuremath{\mathsf{TSR}}$ estimator becomes equivalent to the naive $\ensuremath{\mathsf{CR}}$ estimator, while in the highly supply-constrained limit ($\lambda/\tau \to \infty$), the naive $\ensuremath{\mathsf{TSR}}$ estimator becomes equivalent to the naive $\ensuremath{\mathsf{LR}}$ estimator.  In particular, using Propositions \ref{prop:Qij_demand_constrained} and \ref{prop:Qij_supp_constrained}, it is straightforward to show that the steady-state naive $\ensuremath{\mathsf{TSR}}$ estimator is unbiased in {\em both} limits; we state this as the following theorem, and omit the proof
\begin{corollary}
\label{cor:naive_TSR}
For each $\lambda/\tau$, consider the $\ensuremath{\mathsf{TSR}}$ design with $a_C$ and $a_L$ defined as in \eqref{eq:aC_aL_naive_TSR}.  Consider a sequence of systems where either $\lambda/\tau \to 0$, or $\lambda/\tau \to \infty$.  Then in either limit:
\[ \widehat{\ensuremath{\mathsf{GTE}}}^{\ensuremath{\mathsf{TSR}}}(\infty | a_C(\lambda/\tau), a_L(\lambda/\tau)) - \ensuremath{\mathsf{GTE}}\to 0. \]
\end{corollary}

Figure \ref{fig:numerics} reveals the steady-state performance of the different naive estimators ($\ensuremath{\mathsf{CR}}$, $\ensuremath{\mathsf{LR}}$, and $\ensuremath{\mathsf{TSR}}$ with the preceding scaling of $a_C$ and $a_L$), as a function of market balance $\lambda/\tau$.  The figure shows the difference between each estimator and $\ensuremath{\mathsf{GTE}}$; the estimators are all upward-biased because of the parameter values chosen, though the qualitative findings in the figure are robust across parameter choices.  As we see, the naive $\ensuremath{\mathsf{TSR}}$ estimator performs well in each asymptotic regime.

We are also led to ask whether we can improve upon the naive $\ensuremath{\mathsf{TSR}}$ estimator when the market is moderately balanced. 
Note that the naive $\ensuremath{\mathsf{TSR}}$ estimator does not explicitly correct for {\em either} the fact that there is interference across listings, or the fact that there is interference across customers. We now suggest a heuristic for correction of these effects that leads to an improved interpolating $\ensuremath{\mathsf{TSR}}$ estimator; this is the fourth estimator that appears in Figure \ref{fig:numerics}, \newgw{that we call `Improved $\ensuremath{\mathsf{TSR}}$'}.  

First, abusing notation, let $\ensuremath{\mathsf{GTE}}^{\ensuremath{\mathsf{CR}}}(T|a_C, a_L)$ denote the estimator in \eqref{eq:naive_CR} using the same terms from a $\ensuremath{\mathsf{TSR}}$ design, and  dividing through by $a_L$ on both terms as normalization.  Similarly abusing notation,  let $\ensuremath{\mathsf{GTE}}^{\ensuremath{\mathsf{LR}}}(T|a_C, a_L)$ denote the estimator in \eqref{eq:naive_CR} using the same terms from a $\ensuremath{\mathsf{TSR}}$ design, and dividing through by $a_C$ on both terms as normalization.  Motivated by these naive estimators, we explicitly consider an interpolation between the $\ensuremath{\mathsf{LR}}$ and $\ensuremath{\mathsf{CR}}$ estimators of the form:
\newgw{\begin{align}
\label{eq:interpolatingTSR}
 \beta \widehat{\ensuremath{\mathsf{GTE}}}^{\ensuremath{\mathsf{CR}}}(T|a_C,a_L)+(1-\beta)\widehat{\ensuremath{\mathsf{GTE}}}^{\ensuremath{\mathsf{LR}}}(T|a_C,a_L)& \\ = \beta \left(\frac{Q_{11}(T | a_C,a_L)}{a_Ca_L} - \frac{Q_{01}(T | a_C,a_L)}{(1-a_C)a_L}\right) &+ (1-\beta) \left(\frac{Q_{11}(T | a_C,a_L)}{a_Ca_L} - \frac{Q_{10}(T | a_C,a_L)}{a_C(1-a_L)}\right). \notag
 \end{align}}
 

Now, consider the quantity $Q_{00}(T|a_C,a_L)/((1-a_C)(1-a_L)) - Q_{10}(T|a_C,a_L)/((1-a_C)a_L)$ in a $\ensuremath{\mathsf{TSR}}$ design. This is the (appropriately normalized) difference between the rate at which control customers book control listings, and the rate at which treatment customers book control listings.   Note that the  difference between control customers and treatment customers is that the latter are exposed to treatment listings, while the former are not. Hence, the difference in steady-state rates of booking among these two groups on {\em control listings} must be driven by the fact that treatment customers substitute booking from control listings to treatment listings (or vice versa). This difference is precisely the ``cannibalization" effect (i.e., interference) that was found in $\ensuremath{\mathsf{LR}}$ designs in the highly demand-constrained regime.

Following from this motivation, we subtract an appropriately weighted ``correction term’’ for the $\ensuremath{\mathsf{LR}}$ design from our interpolating $\ensuremath{\mathsf{TSR}}$ estimator in \eqref{eq:interpolatingTSR}. 
(This argument is of course heuristic: it ignores dynamic effects in regimes of intermediate market balance.)
Using a symmetric argument we also subtract an appropriately weighted correction term associated to interference across customers in a $\ensuremath{\mathsf{CR}}$ design:  $Q_{00}(T|a_C,a_L)/((1-a_L)(1-a_C))-Q_{01}(T|a_C,a_L)/((1-a_C)a_L)$.  (Similar correction terms were also studied in \cite{bajari2019double}; see the related work for further details on this work.)
We weight these 
correction terms in a market-balance-dependent fashion, based on the direction of market balance in which we have seen that the respective interference grows.  Combining these insights, for $\beta \in (0,1)$ our improved $\ensuremath{\mathsf{TSR}}$ estimator is given by:
\begin{multline}
\label{eq:TSRI}
\widehat{\ensuremath{\mathsf{GTE}}}^{\ensuremath{\mathsf{TSRI}}}(T|a_C, a_L) = \\
\beta \left[\frac{Q_{11}(T | a_C,a_L)}{a_Ca_L} - \frac{Q_{01}(T | a_C,a_L)}{(1-a_C)a_L}-(1-\beta)\left(\frac{Q_{00}(T|a_C, a_L)}{(1-a_C)(1-a_L)}-\frac{Q_{01}(T|a_C, a_L)}{(1-a_C)a_L}\right) \right] \\
+(1- \beta) \left[ \frac{Q_{11}(T | a_C,a_L)}{a_Ca_L} - \frac{Q_{10}(T | a_C,a_L)}{a_C(1-a_L)}-\beta\left(\frac{Q_{00}(T|a_C,a_L)}{(1-a_C)(1-a_L)}-\frac{Q_{10}(T|a_C,a_L)}{a_C(1-a_L)}\right) \right],
\end{multline}
Given market balance $\lambda/\tau$, we set $\beta=e^{-\lambda/\tau}$, and we choose $a_C$ and $a_L$ as in \eqref{eq:aC_aL_naive_TSR}.

In the limit where $\lambda/\tau \to 0$, note that $\widehat{\ensuremath{\mathsf{GTE}}}^{\ensuremath{\mathsf{TSRI}}} (T|a_C(\lambda/\tau),a_L(\lambda/\tau))\to\widehat{\ensuremath{\mathsf{GTE}}}^{\ensuremath{\mathsf{CR}}}(T|\b{a}_C)$ as expected.  Similarly, in the limit where $\lambda/\tau \to \infty$, we have $\widehat{\ensuremath{\mathsf{GTE}}}^{\ensuremath{\mathsf{TSRI}}} (T|a_C(\lambda/\tau),a_L(\lambda/\tau))\to\widehat{\ensuremath{\mathsf{GTE}}}^{\ensuremath{\mathsf{LR}}}(T|\b{a}_L)$.  In particular, it is straightforward to show as a result that this new estimator is also unbiased in {\em both} the highly demand-constrained and highly supply-constrained regimes.  In these limits, the correction terms play no role.  However, for moderate values of market balance,  both the cannibalization correction terms kick in, \newgw{improving over naive $\ensuremath{\mathsf{TSR}}$ as seen in the figure. As we mentioned, we leave the study of other $\ensuremath{\mathsf{TSR}}$ designs and estimators that may perform even better for future research.}

\gw{Figure 1: should we normalize by GTE as in the next figure?  Can we provide an appendix with more results to give more sense of robustness?}


\begin{figure}
    \centering
    \begin{tabular}{c c}
     \includegraphics[width=0.5\textwidth]{estimator_minus_gte_lambdas.pdf}
    &  
    \includegraphics[width=0.5\textwidth]{transient_dynamics_med_lambda.pdf}
    \end{tabular}
    \caption{Left: Difference between estimator and $\ensuremath{\mathsf{GTE}}$ in steady state.  We consider variation in $\lambda/\tau$ by fixing $\tau = 1$ and varying $\lambda$; analogous results are obtained if $\lambda$ is fixed and $\tau$ is varied.  Right: transient behavior of estimators. Again we fix $\tau = 1$.  For both plots, we set $\epsilon_{\gamma,i}=1, \alpha_{\gamma,i}=0.5$ for all $\gamma, i$. In the $\ensuremath{\mathsf{CR}}$ design, $a_C=1/2$. In the $\ensuremath{\mathsf{LR}}$ design, $a_L=1/2$. There are three listing types and two customer types.
   
    Utilities for $\gamma_1$ are $v_{\gamma_1}(\theta_1)= 1.5, v_{\gamma_1}(\theta_2)=3 , v_{\gamma_1}(\theta_3)=6.
    \tilde{v}_{\gamma_1}(\theta_1)= 1,
    \tilde{v}_{\gamma_1}(\theta_2)=8,
    \tilde{v}_{\gamma_1}(\theta_3)=12.$
    Utilities for $\gamma_2$ are $v_{\gamma_2}(\theta_1= 1, v_{\gamma_2}(\theta_2)=2 , v_{\gamma_2}(\theta_3)=4.
    \tilde{v}_{\gamma_2}(\theta_1)= 0.5,
    \tilde{v}_{\gamma_2}(\theta_2)=6,
    \tilde{v}_{\gamma_2}(\theta_3)=8$.}
    \label{fig:numerics}
\end{figure}



\subsection{Highly demand-constrained markets}

Consider a hypothetical limit where each listing becomes instantly available again after being booked (i.e., $\tau \to \infty$ but $\lambda$ remains fixed).  This is the demand-constrained extreme, where capacity constraints on listings become irrelevant.  Note that on arrival of a customer, both listings are always available, and therefore, included in the consideration set.  In this limit, observe that the steady-state rate at which customers book listing $\ell = 1,2$ becomes:
\begin{equation}
\label{eq:example_rate_dc}
\frac{\lambda v}{\epsilon + 2 v}.
\end{equation}
(The factor 2 appears in the denominator as there are two listings.)  Since the intervention changes $v$ to $\tilde{v}$, the $\ensuremath{\mathsf{GTE}}$ is:
\[ \ensuremath{\mathsf{GTE}} = \frac{2 \lambda \tilde{v}}{\epsilon + 2 \tilde{v}} - \frac{2\lambda v}{\epsilon + 2v}. \]

Now suppose we consider a $\ensuremath{\mathsf{CR}}$ design that randomizes a fraction $a_C$ of arriving customers to treatment.  Observe that in this demand-constrained extreme, every arriving control (resp., treatment) customer sees the full global control (resp., treatment) market condition; there is no dynamic influence of one customer's booking behavior on any other customer.  This suggests the naive $\ensuremath{\mathsf{CR}}$ estimator should correctly recover the global treatment effect.  Indeed, the steady-state naive $\ensuremath{\mathsf{CR}}$ estimator becomes:
\[ \widehat{\ensuremath{\mathsf{GTE}}}^{\ensuremath{\mathsf{CR}}}(\infty | a_C) = \frac{1}{a_C} \frac{2 a_C \lambda \tilde{v}}{\epsilon + 2 \tilde{v}} - \frac{1}{1-a_C} \frac{2 (1-a_C) \lambda v}{\epsilon + 2 v} = \ensuremath{\mathsf{GTE}}. \]
In other words, the naive $\ensuremath{\mathsf{CR}}$ estimator is perfectly {\em unbiased}.  

On the other hand, consider a $\ensuremath{\mathsf{LR}}$ design where listing 1 is (randomly) assigned to treatment, and listing 2 is (randomly) assigned to control.  In this design, the steady-state naive $\ensuremath{\mathsf{LR}}$ estimator (with $a_L = 1/2$) becomes:
\[ \widehat{\ensuremath{\mathsf{GTE}}}^{\ensuremath{\mathsf{LR}}}(\infty | a_L) =  \frac{1}{a_L} \frac{\lambda \tilde{v}}{\epsilon + v + \tilde{v}} - \frac{1}{1-a_L} \frac{\lambda v}{\epsilon + v + \tilde{v}}. \]
It is clear that in general this will {\em not} be equal to the $\ensuremath{\mathsf{GTE}}$, because there is {\em interference} between the two listings: {\em every} arriving customer sees a market environment that is neither quite global treatment nor global control, and the estimates reflect this imperfection.  Even with immediate replenishment, treatment listings compete for customers and ``cannibalize" bookings from control listings,
causing the naive $\ensuremath{\mathsf{LR}}$ estimator to be biased.  (Note that such a violation would arise for virtually any reasonable choice model that could be considered.) 

\subsection{Highly supply-constrained markets}

Now we consider the opposite extreme, where the market is heavily supply constrained; in particular, we consider the hypothetical limit where $\lambda \to \infty$ but $\tau$ remains fixed.  Now in this case, note that a listing that becomes available will {\em nearly instantaneously} be booked; therefore, virtually every arriving customer will find at most one of the two listings available, and their decision of whether to book will be entirely determined by comparison of that available listing against the outside option.  In particular, as a result when $\lambda \to \infty$ the steady-state rate at which listing $\ell = 1,2$ is booked approaches $\tau$.

We thus require a more refined estimate of this booking rate as $\lambda \to \infty$.  Suppose $\lambda$ is large, and suppose listing $\ell$ becomes available.  Based on the intuition above, we make the approximation that the listing will be considered in isolation by a succession of customers until it is booked.  Customers arrive at rate $\lambda$, and book an available listing with probability $v/(\epsilon + v)$; in other words, in this regime listings compete only with the outside option, and not with each other.  Therefore the mean time until such a booking occurs is $(\epsilon + v)/\lambda v$; and once booked, the listing remains booked for mean time $1/\tau$, at which time it becomes available again.  Therefore for large $\lambda$, the long-run average rate at which a listing $\ell = 1,2$ is booked is approximately:
\begin{equation}
\label{eq:example_rate_sc}
\left( \frac{\epsilon + v}{\lambda v} + \frac{1}{\tau}\right)^{-1}.
\end{equation}
As expected, this rate approaches $\tau$ as $\lambda \to \infty$.  The $\ensuremath{\mathsf{GTE}}$ is thus:
\[ \ensuremath{\mathsf{GTE}} = 2\left( \frac{\epsilon + \tilde{v}}{\lambda \tilde{v}} + \frac{1}{\tau}\right)^{-1} - 2 \left( \frac{\epsilon + v}{\lambda v} + \frac{1}{\tau}\right)^{-1}. \]

With this observation in hand, suppose we again consider the same $\ensuremath{\mathsf{LR}}$ design where listing 1 is (randomly) assigned to treatment, and listing 2 is (randomly) assigned to control.  Since in \eqref{eq:example_rate_sc} there is no influence of one listing on the other, observe that the naive $\ensuremath{\mathsf{LR}}$ estimator (with $a_L = 1/2$) becomes:
\[ \widehat{\ensuremath{\mathsf{GTE}}}^{\ensuremath{\mathsf{LR}}}(\infty | a_L) = \frac{1}{a_L} \left( \frac{\epsilon + \tilde{v}}{\lambda \tilde{v}} + \frac{1}{\tau}\right)^{-1} - \frac{1}{1-a_L}\left( \frac{\epsilon + v}{\lambda v} + \frac{1}{\tau}\right)^{-1} = \ensuremath{\mathsf{GTE}}. \]
In other words, the naive $\ensuremath{\mathsf{LR}}$ estimator is perfectly {\em unbiased}.  This is intuitive: in the limit where $\lambda$ is large, since listings do not compete with each other for bookings, there is no interference when we implement the $\ensuremath{\mathsf{LR}}$ design.


On the other hand, consider the naive $\ensuremath{\mathsf{CR}}$ design where a fraction $a_C$ of arriving customers are randomized to treatment.  In this case we wish to establish the rate at which bookings are made by treatment and control customers respectively.  Suppose listing $\ell$ was occupied by a treatment customer, and becomes available.  Define:
\[ \zeta(a_C) = a_C \tilde{v}/(\epsilon + \tilde{v}) + (1 - a_C)v/(\epsilon + v). \]
This is the probability an arriving customer books the available listing.  Customers arrive at rate $\lambda$, so a mean time $1/(\lambda \zeta(a_C))$ elapses until a booking is made; the listing then remains occupied for mean time $1/\tau$.  Hence, the long-run average rate at which bookings of listing $\ell$ occur is given by $(1/(\lambda \zeta(a_C))+1/\tau)^{-1}$. 
Now, conditional on a booking, the booking was made by a treatment customer with probability:
\[ \eta(a_C) = \frac{1}{\zeta(a_C)} \frac{a_C \tilde{v}}{\epsilon + \tilde{v}}. \]
Hence, 
Thus the long-run average rate at which listing $\ell$ is booked by treatment customers is:
\[ \eta(a_C) \cdot \left(\frac{1}{\lambda \zeta(a_C)} + \frac{1}{\tau}\right)^{-1} =  \left( \frac{\epsilon + \tilde{v}}{a_C \lambda \tilde{v}} + \frac{1}{\eta(a_C) \tau} \right)^{-1}. \] 
We can use the same logic for the booking rate of control customers, and so we find the naive $\ensuremath{\mathsf{CR}}$ estimator is:
\[ \widehat{\ensuremath{\mathsf{GTE}}}^{\ensuremath{\mathsf{CR}}}(\infty | a_C) = 
\frac{2}{a_C} \left( \frac{\epsilon + \tilde{v}}{a_C \lambda \tilde{v}} + \frac{1}{\eta(a_C) \tau}\right)^{-1} - 
\frac{2}{1-a_C}\left( \frac{\epsilon + v}{(1-a_C)\lambda v} + \frac{1}{(1 - \eta(a_C))\tau}\right)^{-1}. \]

In general, this estimator will be {\em biased}, i.e., not equal to $\ensuremath{\mathsf{GTE}}$.  The issue is that in this case, customers have a {\em dynamic} influence on each other across the treatment groups: when a listing becomes available, whether or not it is available for booking by a subsequent control customer depends on whether or not a treatment customer had previously booked the listing. In this case, customers compete among each other for listings. This interference across customer groups leads to the biased expression for the naive $\ensuremath{\mathsf{CR}}$ estimator.

We note that the $\ensuremath{\mathsf{GTE}}$ and naive $\ensuremath{\mathsf{LR}}$ estimators converge to zero in the limit where $\lambda \to \infty$; this is because the booking rate of each listing becomes $\tau$ in this limit.  The naive $\ensuremath{\mathsf{CR}}$ estimator does {\em not} converge to zero in general, however, as $\lambda \to \infty$, because in general 
$\eta(a_C) = (1/\zeta(a_C)) \cdot (a_C \tilde{v})/(\epsilon + \tilde{v}) \neq  (1/\zeta(a_C)) \cdot ((1-a_C) {v})/(\epsilon + {v}) = 1-\eta(a_C)$.  For example, if $\tilde{v} > v$ and $a_C = 1/2$, these expressions reveal that treatment customers book more often than control customers.



\subsection{Discussion: Violation of SUTVA}

Our simple example illustrates that the naive $\ensuremath{\mathsf{LR}}$ estimator is biased when $\lambda \to 0$, and unbiased when $\lambda \to \infty$; and the naive $\ensuremath{\mathsf{CR}}$ estimator is unbiased when $\lambda \to 0$, while it is biased when $\lambda \to \infty$.
These findings can be interpreted through the lens of the classical potential outcomes model; an important result from this literature is that when the {\em stable unit treatment value assumption} (SUTVA) holds, then naive estimators of the sort we consider will be unbiased for the true treatment effect.  SUTVA requires that the treatment condition of units other than a given customer or listing should not influence the potential outcomes of that given customer or listing.  The discussion above illustrates that in the limit where $\lambda \to 0$, there is no interference across customers in the $\ensuremath{\mathsf{CR}}$ design; this is why the naive $\ensuremath{\mathsf{CR}}$ estimator is unbiased.  Similarly, in the limit where $\lambda \to \infty$, there is no interference across listings in the $\ensuremath{\mathsf{LR}}$ design; this is why the naive $\ensuremath{\mathsf{LR}}$ estimator is unbiased.  On the other hand, the cases where each estimator is biased involve interference across experimental units.





\subsection{Simulation details}

For every setting, we run 500 simulations with $N=5000$. We fix $\alpha=1$, $\epsilon=1$, and $\tau=1$ across all settings and for each we consider three levels of demand: $\lambda= 0.1, 1, 10$. The simulation runs for $T_1$ time units. We drop the burn in time and calculate the value of the estimator on the time interval $[T_0, T_1]$ for some $0 < T_0 < T_1$. We set $T_0=5$ and $T_1=25$. 

We further scale time by $\min\{\lambda, \tau\}$ so that the number of ``events'' (i.e., bookings) that occur in the time interval is consistent across different values of $\lambda$ and $\tau$. More specifically, for a given $(\lambda, \tau)$ pair, we calculate the value of the estimator on the time interval $[T_0 / \min\{\lambda, \tau\}, \  T_1/\min\{\lambda, \tau\}]$. 
To see why this is necessary, note that in the case where there is no outside option and all customers book a listing if one is available, the rate of bookings is $\min(\lambda, \tau)$. We heuristically rescale time so that if we removed the effect of choice set dynamics and utilities and all customers book any available listing, the number of bookings would be consistent across market balance levels. 
This rescaling allows us to achieve similar precision in our estimates of bias across different levels of market balance; that is, under this rescaling the size of the standard error of the estimates is similar across different levels of market balance. Because of this rescaling, it is difficult to compare how standard errors of the estimators change when we fix the time horizon and change market balance. We suggest that the reader use the standard error figures to compare the standard error of different estimators \textit{within} the same level of market balance. 




For the base setting shown in Section \ref{sec:variance} with homogeneous listings and customers, a customer has utility $v = 0.315$ for a control listing and $\tilde{v}=0.3937$ for a treatment listing, corresponding to a mean field steady state booking probability of 20 percent in the global control model and 23 percent in the global treatment model. This change corresponds to a 25 percent increase in the utility, due to treatment.
Unless otherwise noted, all sets of market parameters are chosen to maintain the 20 and 23 percent booking probabilities in global control and treatment, respectively. 

We fix $a_C=0.5$ in the $\ensuremath{\mathsf{CR}}$ experiments and $a_L=0.5$ in the $\ensuremath{\mathsf{LR}}$ experiments. For the $\ensuremath{\mathsf{TSR}}$ experiments, we vary $a_C$ and $a_L$ as defined in Section \ref{ssec:TSR_est}. For a general $\ensuremath{\mathsf{TSR}}$ experiment with randomization parameters $(a_C, a_L)$, we simulate a completely randomized design on the listing side, fixing $\lfloor N \cdot a_L \rfloor$ listings in treatment and $\lceil N \cdot (1-a_L) \rceil$ listings in control. Since customers arrive over time, we cannot run a completely randomized design and so we simulate Bernoulli randomization on the customers, randomizing each customer to treatment independently with probability $a_C$. 

For each statistic (bias, standard error, and RMSE), bootstrapped 95th percentile confidence intervals are presented. In each setting, we re-sample 500 simulation runs, with replacement, from the original set of 500 simulations. We calculate the value of the bias, standard error, and RMSE for each estimator and present the 95th percentile intervals of each. 

\subsection{Market scenarios}
\label{ssec:robustness_market_scenarios}

In this section, we analyze the effect of changes in customers' choice set parameters on the behavior of the estimators, while holding the market balanced fixed with $\lambda = \tau = 1$. In all of these scenarios, $\ensuremath{\mathsf{TSRI\text{-}2}}$ has the lowest bias and the highest variance. $\ensuremath{\mathsf{TSRI\text{-}2}}$ is the estimator that minimizes $\mathrm{RMSE}$ in these simulations, though this depends on the relative magnitude of the bias and standard error, as a result of the size of the market. For a larger market, the standard error is smaller the magnitude of the bias, and vice versa for a smaller market. 

\textbf{Varying average utility.}
In addition to the homogeneous system with one customer type and one listing type presented in Figure \ref{fig:simulations_hom}, we consider two additional settings, again in a homogeneous system, but now scaling the utility $v$. We fix the effect of the intervention such that the lift in utility $\tilde{v}/v = 1.25$ is constant in all three settings, and consider settings with smaller $v$ and larger $v$. Rescaling $v$ will change the steady state booking probabilities in global treatment and control so that we no longer have 23 percent of booking in global treatment and 20 percent in global control. 

Note that changes to $\alpha$ and/or $\epsilon$ can be equivalently viewed as a rescaling of the utility, so changing the utility also allows us to explore the effect of changes in $\alpha$ or $\epsilon$ as well.

\begin{itemize}
    \item Low utility: $v = 0.155, \tilde{v} = 0.1938$. 
    \item Medium utility: $v = 0.315, \tilde{v} = 0.3937$.
    \item High utility: $v = 0.62, \tilde{v} = 0.775$.
\end{itemize}

Results are presented in Figure \ref{fig:robustness_vary_avg_utility}. As the average utility increases (and so does the steady state booking probability of arriving customers), the bias for $\ensuremath{\mathsf{CR}}, \ensuremath{\mathsf{LR}}, \ensuremath{\mathsf{TSR}}-naive$ and $\ensuremath{\mathsf{TSRI\text{-}1}}$ all increase, whereas the bias $\ensuremath{\mathsf{TSRI\text{-}2}}$ is largely consistent across different levels of utility. The standard errors of the estimators do not change notably as we vary the average utility. A consequence of the change in bias is that the $\mathrm{RMSE}$ minimizing estimator depends on the utility level, with $\ensuremath{\mathsf{TSRI\text{-}2}}$ having the highest $\mathrm{RMSE}$ for $v=0.15$ but the smallest for $v=0.62$. 

\begin{figure}
    \centering
    \begin{tabular}{l l }
        \includegraphics[height=.26\textwidth]{figs_sims/robustness_across_param/mf_varying_utility_mod.png}
        &
        \includegraphics[height=.26\textwidth]{figs_sims/robustness_across_param/sims_bias_varying_utility_lam_1_error_bars_mod}
        \\
        \includegraphics[height=.26\textwidth]{figs_sims/robustness_across_param/sims_se_varying_utility_lam_1_error_bars_mod}
        &
        \includegraphics[height=.26\textwidth]{figs_sims/robustness_across_param/sims_rmse_varying_utility_lam_1_error_bars_mod}
    \end{tabular}
    \caption{(Varying average utility.) Results in balanced market with $\lambda=\tau=1$. 
    Top left: Bias of each estimator in the mean field model (normalized by GTE). 
    Top right: Bias of each estimator in simulations, averaged across 500 runs (normalized by GTE). 
    Bottom left: Standard error of estimates, calculated across 500 runs (normalized by GTE). 
    Bottom right: $\mathrm{RMSE}$ of the estimates, calculated across $500$ runs (normalized by GTE). 
    }
    \label{fig:robustness_vary_avg_utility}
\end{figure}





\textbf{Heterogeneity of customers.} There is one listing type $\theta$ and two customer types $\gamma_1$, $\gamma_2$. We fix the size of the treatment utility increase such that $\tilde{v}_{\gamma_1}(\theta) = 1.25 \cdot  v_{\gamma_1}(\theta)$ and $\tilde{v}_{\gamma_2}(\theta) = 1.25 \cdot v_{\gamma_2}(\theta)$. We additionally fix the steady state booking probabilities in global control and global treatment to be 20 percent and 23 percent, respectively.

Customers of type $\gamma_1$ have higher utility for the listing than customers of type $\gamma_2$. We vary the heterogeneity of the customers by varying the ratio $v_{\gamma_2}(\theta)/ v_{\gamma_1}(\theta)$.
\begin{itemize}
    \item Homogeneous: $v_{\gamma_2}(\theta)/ v_{\gamma_1}(\theta)=1$, with $v_{\gamma_1}(\theta) = 0.315,  v_{\gamma_2}(\theta) = 0.315$. 
    \item Low level of heterogeneity (Het-L): $v_{\gamma_2}(\theta)/ v_{\gamma_1}(\theta)=3$, with $v_{\gamma_1}(\theta) = 0.17,  v_{\gamma_2}(\theta) = 0.51$. 
    \item High level of heterogeneity (Het-H): $v_{\gamma_2}(\theta)/ v_{\gamma_1}(\theta)=3.83$, with $v_{\gamma_1}(\theta) = 0.12,  v_{\gamma_2}(\theta) = 0.46$. 
\end{itemize}

Results are presented in Figure \ref{fig:robustness_vary_customer_het}. As heterogeneity of the customers increases, the $\ensuremath{\mathsf{CR}}$ estimator has similar levels of bias, while $\ensuremath{\mathsf{LR}}, \ensuremath{\mathsf{TSRN}}$, and $\ensuremath{\mathsf{TSRI\text{-}1}}$ slightly increase, though not as appreciably as the increase seen when varying the average utility level. In all cases, $\ensuremath{\mathsf{TSRI\text{-}2}}$ has the lowest bias and highest standard error. In these simulations, $\ensuremath{\mathsf{TSRI\text{-}2}}$ is the estimator that minimizes $\mathrm{RMSE}$, although this can change depending on the size of the market and the relative sizes of the bias and the standard error. 


\begin{figure}
    \centering
    \begin{tabular}{l l }
        \includegraphics[height=.26\textwidth]{figs_sims/robustness_across_param/mf_varying_customer_het_mod.png}
        &
        \includegraphics[height=.26\textwidth]{figs_sims/robustness_across_param/sims_bias_varying_customer_het_lam_1_error_bars_mod}
        \\
        \includegraphics[height=.26\textwidth]{figs_sims/robustness_across_param/sims_se_varying_customer_het_lam_1_error_bars_mod}
        & 
        \includegraphics[height=.26\textwidth]{figs_sims/robustness_across_param/sims_rmse_varying_customer_het_lam_1_error_bars_mod}
    \end{tabular}
    \caption{(Varying heterogeneity of customers.) Results in balanced market with $\lambda=\tau=1$.
    Top left: Bias of each estimator in the mean field model (normalized by GTE). 
    Top right: Bias of each estimator in simulations, averaged across 500 runs (normalized by GTE). 
    Bottom left: Standard error of estimates, calculated across 500 runs (normalized by GTE). 
    Bottom right: $\mathrm{RMSE}$ of the estimates, calculated across $500$ runs (normalized by GTE). 
    }
    \label{fig:robustness_vary_customer_het}
\end{figure}








\textbf{Heterogeneity of listings.} There are two listing types $\theta_1$ and $\theta_1$ and one customer type $\gamma$. We fix the treatment effect so that $\tilde{v}(\theta) = 1.25 \cdot v(\theta)$. We additionally fix the steady state booking probabilities in global control and global treatment to be 20 percent and 23 percent, respectively.

We vary the heterogeneity of the listings by letting $v(\theta_2)>v(\theta_1)$ and varying the ratio $v(\theta_2) / v(\theta_1)$. 
\begin{itemize}
    \item Homogeneous: $v(\theta_2) / v(\theta_1) = 1$, with $v(\theta_1) = v(\theta_2) = 0.315$.
    \item Low level of heterogeneity (Het-L): $v(\theta_2) / v(\theta_1) = 1.6$, with $v(\theta_1) = 0.25, v(\theta_2) = 0.4$.
    \item High level of heterogeneity (Het-H): $v(\theta_2) / v(\theta_1)=6$, with $v(\theta_1) = 0.1, v(\theta_2) = 0.6$. 
\end{itemize}

Results are presented in Figure \ref{fig:robustness_vary_list_het}. As heterogeneity of the listings increases, the bias of the $\ensuremath{\mathsf{CR}}$ estimator increases significantly, the $\ensuremath{\mathsf{TSRN}}$ estimator increases to a lesser extent, and the $\ensuremath{\mathsf{LR}}$, $\ensuremath{\mathsf{TSRI\text{-}1}}$, and $\ensuremath{\mathsf{TSRI\text{-}2}}$ estimators remain roughly consistent. In all cases, $\ensuremath{\mathsf{TSRI\text{-}2}}$ has the lowest bias and highest standard error. In these simulations, $\ensuremath{\mathsf{TSRI\text{-}2}}$ is the estimator that minimizes $\mathrm{RMSE}$, although this can change depending on the size of the market and the relative sizes of the bias and the standard error. 

\begin{figure}
    \centering
    \begin{tabular}{l l }
        \includegraphics[height=.26\textwidth]{figs_sims/robustness_across_param/mf_varying_list_het_mod.png}
        &
        \includegraphics[height=.26\textwidth]{figs_sims/robustness_across_param/sims_bias_varying_list_het_lam_1_error_bars_mod}
        \\
        \includegraphics[height=.26\textwidth]{figs_sims/robustness_across_param/sims_se_varying_list_het_lam_1_error_bars_mod}
        & 
        \includegraphics[height=.26\textwidth]{figs_sims/robustness_across_param/sims_rmse_varying_list_het_lam_1_error_bars_mod}
    \end{tabular}
    \caption{(Varying heterogeneity of listings.) Results in balanced market with $\lambda=\tau=1$.
    Top left: Bias of each estimator in the mean field model (normalized by GTE). 
    Top right: Bias of each estimator in simulations, averaged across 500 runs (normalized by GTE). 
    Bottom left: Standard error of estimates, calculated across 500 runs (normalized by GTE). 
    Bottom right: $\mathrm{RMSE}$ of the estimates, calculated across $500$ runs (normalized by GTE). 
    }
    \label{fig:robustness_vary_list_het}
\end{figure}









\textbf{Heterogeneity of treatment effect.}
We consider the impact of heterogeneous treatment effects on the performance of the estimators. We restrict attention to where the treatment increases the utility, but the size of the increase can differ across the market. 

There are two listing types $\theta_1$ and $\theta_2$ and one customer type $\gamma$. We fix the customer's preferences pre-treatment such that the customer prefers $\theta_2$ to $\theta_1$ pre-treatment, with $v_\gamma(\theta_2)/v_\gamma(\theta_1) = 1.3$.  We compare three settings, one where the treatment effect has the same multiplicative lift on both listing types, one where the treatment amplifies the existing preference order, and one where the treatment reverses the existing preference order. 

We fix the steady state booking probabilities and global control and global treatment to be 20 percent and 23 percent, respectively. In all of the scenarios, we have $v(\theta_1) = 0.27, v(\theta_2)=0.351$.

\begin{itemize}
    \item Multiplicative: $\tilde{v}(\theta_1)=0.3375, \tilde{v}(\theta_2)=0.4388$.
    \item Heterogeneous treatment effects - amplify (HTE-amp): $\tilde{v}(\theta_1)=0.2727, \tilde{v}(\theta_2)=0.5265$.
    \item Heterogeneous treatment effects - reverse (HTE-rev): $\tilde{v}(\theta_1)=0.432, \tilde{v}(\theta_2)=0.355$.
\end{itemize}

Results are presented in Figure \ref{fig:robustness_vary_htes}. We find that when the treatment effect amplifies the existing preferences, the $\ensuremath{\mathsf{CR}}$ estimator has a larger bias. This is likely an artifact of the increase in listing heterogeneity after treatment (see discussion on varying listing heterogeneity and Figure \ref{fig:robustness_vary_list_het}). The bias is similar in the two settings with a multiplicative treatment effect and heterogeneous treatment effects that reverse the control preference order. In all cases, $\ensuremath{\mathsf{TSRI\text{-}1}}$ has the lowest bias, highest standard error, and lowest $\mathrm{RMSE}$. 


\begin{figure}
    \centering
    \begin{tabular}{l l}
        \includegraphics[height=.26\textwidth]{figs_sims/robustness_across_param/mf_varying_htes_mod.png}
        &
        \includegraphics[height=.26\textwidth]{figs_sims/robustness_across_param/sims_bias_varying_htes_lam_1_error_bars_mod}
        \\
        \includegraphics[height=.26\textwidth]{figs_sims/robustness_across_param/sims_se_varying_htes_lam_1_error_bars_mod}
        & 
        \includegraphics[height=.26\textwidth]{figs_sims/robustness_across_param/sims_rmse_varying_htes_lam_1_error_bars_mod}
    \end{tabular}
    \caption{(Varying heterogeneity of treatment lift on utilities.) Results in balanced market with $\lambda=\tau=1$. 
    Top left: Bias of each estimator in the mean field model (normalized by GTE). 
    Top right: Bias of each estimator in simulations, averaged across 500 runs (normalized by GTE). 
    Bottom left: Standard error of estimates, calculated across 500 runs (normalized by GTE). 
    Bottom right: $\mathrm{RMSE}$ of the estimates, calculated across $500$ runs (normalized by GTE).
    }
    \label{fig:robustness_vary_htes}
\end{figure}


\subsection{Robustness with varying market balance}
\label{ssec:robustness_vary_lambda}


We now replicate Figures \ref{fig:numerics_homogeneous} and \ref{fig:simulations_hom} for different market settings, verifying the results we present in the main text regarding the dependence of the estimators on market balance. In Section \ref{ssec:robustness_market_scenarios}, we consider four scenarios at a fixed market balance, by varying average utility, customer heterogeneity, listing heterogeneity, and treatment effect heterogeneity. In this section, we choose one representative set of parameters from each scenario and show how the performance of the estimators change as we change market balance. In particular, we show that the behavior of the estimators in a large market setting is still similar to the behavior obtained in the mean field limit, even in the presence of heterogeneity in the market. 

The representative settings shown are low utility (varying average utility), high level of heterogeneity (heterogeneity of customers), high level of heterogeneity (heterogeneity of listings), and heterogeneous treatment effects amplifying existing preference (heterogeneity of treatment effects). The parameters are as defined in Section \ref{ssec:robustness_market_scenarios}. One might suspect that these settings, either with a low booking probability or high levels of heterogeneity, are the ones most likely to differ from the mean field model. We show, however, that the findings in the main text are indeed robust.

Across our range of simulations, we see qualitatively similar behavior to our findings in the main text: the naive $\ensuremath{\mathsf{CR}}$ estimator has lower bias for small $\lambda/\tau$ and the naive $\ensuremath{\mathsf{LR}}$ estimator has lower bias for large $\lambda/\tau$, while the $\ensuremath{\mathsf{TSR}}$ estimators interpolate between the two.  The $\ensuremath{\mathsf{TSR}}$ estimators offer these bias reductions at the cost of higher variance.   Further, the bias of $\ensuremath{\mathsf{CR}}$ is higher at larger $\lambda/\tau$, and the bias of $\ensuremath{\mathsf{LR}}$ is higher at smaller $\lambda/\tau$.

We note that although the qualitative findings are similar, the bias in the simulations is not identical to the bias in the mean field model, especially at the smaller and larger values of $\lambda$.  We conjecture that this is due to the fact that stochastic effects matter in these extremes.  When $\lambda$ is small, since $T$ is fixed, we see a (relatively) smaller number of customer arrivals in the same time horizon; indeed, this is why standard errors are higher overall in this setting.  On the other hand, when $\lambda$ is large, then (relatively) few listings will be available.  In this setting, if the steady state number of available listings is very small relative to $N$, then our mean field approximation will begin to be less accurate, even though the qualitative behavior is similar. The discrepancies may be larger with the $\ensuremath{\mathsf{TSR}}$ estimators, since there are even fewer interactions happening in each of the four cells. 


\begin{figure}
    \centering
    \begin{tabular}{l l }
        \includegraphics[height=.26\textwidth]{figs_sims/robustness_across_param/avg_v_small_mf_bias_vary_lambda_mod.png}
        &
        \includegraphics[height=.26\textwidth]{figs_sims/robustness_across_param/avg_v_small_bias_simulations_error_bar_mod.png}
        \\
        \includegraphics[height=.26\textwidth]{figs_sims/robustness_across_param/avg_v_small_se_simulations_error_bar_mod.png}
        & 
        \includegraphics[height=.26\textwidth]{figs_sims/robustness_across_param/avg_v_small_rmse_simulations_error_bar_mod.png}
    \end{tabular}
    \caption{(Varying market balance - small average utility.)
    Top left: Bias of each estimator in the mean field model (normalized by GTE). 
    Top right: Bias of each estimator in simulations, averaged across 500 runs (normalized by GTE). 
    Bottom left: Standard error of estimates, calculated across 500 runs (normalized by GTE). 
    Bottom right: $\mathrm{RMSE}$ of the estimates, calculated across $500$ runs (normalized by GTE). 
    }
    \label{fig:robustness_vary_lambda_small_utility}
\end{figure}


\begin{figure}
    \centering
    \begin{tabular}{l l }
        \includegraphics[height=.26\textwidth]{figs_sims/robustness_across_param/het_of_customers_large_mf_bias_vary_lambda_mod.png}
        &
        \includegraphics[height=.26\textwidth]{figs_sims/robustness_across_param/het_of_customers_large_bias_simulations_error_bar_mod.png}
        \\
        \includegraphics[height=.26\textwidth]{figs_sims/robustness_across_param/het_of_customers_large_se_simulations_error_bar_mod}
        & 
        \includegraphics[height=.26\textwidth]{figs_sims/robustness_across_param/het_of_customers_large_rmse_simulations_error_bar_mod}
    \end{tabular}
    \caption{(Varying market balance - large heterogeneity between customers.)
    Top left: Bias of each estimator in the mean field model (normalized by GTE). 
    Top right: Bias of each estimator in simulations, averaged across 200 runs (normalized by GTE). 
    Bottom left: Standard error of estimates, calculated across 200 runs (normalized by GTE). 
    Bottom right: $\mathrm{RMSE}$ of the estimates, calculated across $200$ runs (normalized by GTE). 
    }
    \label{fig:robustness_vary_lambda_heterogeneous_customers}
\end{figure}


\begin{figure}
    \centering
    \begin{tabular}{l l }
        \includegraphics[height=.26\textwidth]{figs_sims/robustness_across_param/het_of_listings_large_mf_bias_vary_lambda_mod.png}
        &
        \includegraphics[height=.26\textwidth]{figs_sims/robustness_across_param/het_of_listings_large_bias_simulations_error_bar_mod}
        \\
        \includegraphics[height=.26\textwidth]{figs_sims/robustness_across_param/het_of_listings_large_se_simulations_error_bar_mod}
        & 
        \includegraphics[height=.26\textwidth]{figs_sims/robustness_across_param/het_of_listings_large_rmse_simulations_error_bar_mod}
    \end{tabular}
    \caption{(Varying market balance - large heterogeneity between listings.)
    Top left: Bias of each estimator in the mean field model (normalized by GTE). 
    Top right: Bias of each estimator in simulations, averaged across 500 runs (normalized by GTE). 
    Bottom left: Standard error of estimates, calculated across 500 runs (normalized by GTE). 
    Bottom right: $\mathrm{RMSE}$ of the estimates, calculated across $500$ runs (normalized by GTE). 
    }
    \label{fig:robustness_vary_lambda_heterogeneous_listings}
\end{figure}




\begin{figure}
    \centering
    \begin{tabular}{l l }
        \includegraphics[height=.26\textwidth]{figs_sims/robustness_across_param/hte_amp_large_mf_bias_vary_lambda_mod.png}
        &
        \includegraphics[height=.26\textwidth]{figs_sims/robustness_across_param/hte_amp_large_bias_simulations_error_bar_mod}
        \\
        \includegraphics[height=.26\textwidth]{figs_sims/robustness_across_param/hte_amp_large_se_simulations_error_bar_mod}
        & 
        \includegraphics[height=.26\textwidth]{figs_sims/robustness_across_param/hte_amp_large_rmse_simulations_error_bar_mod}
    \end{tabular}
    \caption{(Varying market balance - heterogeneous treatment effects amplifying global control preferences.)
    Top left: Bias of each estimator in the mean field model (normalized by GTE). 
    Top right: Bias of each estimator in simulations, averaged across 200 runs (normalized by GTE). 
    Bottom left: Standard error of estimates, calculated across 200 runs (normalized by GTE). 
    Bottom right: $\mathrm{RMSE}$ of the estimates, calculated across $200$ runs (normalized by GTE). 
    }
    \label{fig:robustness_vary_lambda_heterogeneous_customers}
\end{figure}


\medskip
\subsection{Modifications of consideration sets.}


Here we consider a modification to the consideration set formation process described in Section \ref{sec:model}. Instead of a customer sampling each listing independently with probability $\alpha$, we now consider a situation where a customer samples a fixed set of $K = 50$ listings into their consideration set, drawn uniformly at random, as long as there are at least $50$ listings available. When there are fewer than $50$ listings, the customer samples all listings in to their consideration set. We note that another scenario to investigate is the one in which listings are drawn proportionally to their utility to the customer, to capture search and recommendation algorithms that recommend listings that the customer is likely to book. We leave deeper investigation of the formation of consideration sets (including sensitivity to the value of $K$) for future work. 

We calibrate the utilities such that the mean field model where customers sample listings into their choice set with probability $K/N = 50/N$ has a steady state booking probability of 20 percent in global control and 23 percent in global treatment, when $\lambda=\tau$. Note that with this change in consideration set formation, we can no longer guarantee that the finite system converges to the mean field model defined in Section \ref{sec:meanfield}. Thus we study these scenarios through simulations in the finite model and present the bias and standard error of the estimators in the simulations (see Figure \ref{fig:choice_set_k_50}).  As noted above, we find qualitatively similar behavior as before.



\begin{figure}[H]
    \centering
    \includegraphics[height=.27\textwidth]{figs_choice_set/fixed_k_50_bias_simulations_error_bar_mod.png}
    \includegraphics[height=.27\textwidth]{figs_choice_set/fixed_k_50_se_simulations_error_bar_mod.png} \\
    \includegraphics[height=.27\textwidth]{figs_choice_set/fixed_k_50_rmse_simulations_error_bar_mod.png}

    \caption{(Fixed size $K = 50$ consideration set.) Top left: Average bias (normalized by $\ensuremath{\mathsf{GTE}}$) of each estimator across 500 runs. 
    Top right: Standard error of estimates, calculated across 500 runs (normalized by $\ensuremath{\mathsf{GTE}}$). 
    Bottom: RMSE of estimates, calculated across 500 runs (normalized by $\ensuremath{\mathsf{GTE}}$).
    There is one customer type and one listing type. Customers have utility $v=6$ for control listings and $\tilde{v}=7.5$ for treatment listings.  
    }
    \label{fig:choice_set_k_50}
\end{figure}





\section{Analysis of bias}
\label{sec:bias}

We now utilize the framework defined to analyze the bias of two common experiment types, $\ensuremath{\mathsf{LR}}$ experiments and $\ensuremath{\mathsf{CR}}$ experiments. 
Recall from Section \ref{sec:intro} that, in a setting where listings are immediately replenished, all customers see the full set of original listings as available. There is no competition between customers but there is still competition between listings, and so, intuitively, we expect $\ensuremath{\mathsf{CR}}$ to be unbiased and $\ensuremath{\mathsf{LR}}$ to be biased. Meanwhile, in a setting where listings remain unavailable for some amount of time, the resulting dynamic linkage across customers creates a bias in $\ensuremath{\mathsf{CR}}$ as well. Now, consider the extreme where the market is highly supply-constrained: most customers who arrive see no available listings, but some customers arrive just as a booked listing becomes available and see a single available listing. Such customers compare the listing against the outside option, but, since no other listings are available, do not compare listings against each other. In this regime, there is no competition across listings but there is competition between customers, and so we expect $\ensuremath{\mathsf{LR}}$ to be unbiased and $\ensuremath{\mathsf{CR}}$ to be biased.

In this section, we formalize this intuition about the behavior of the estimators in the extremes of market balance. We establish two key theoretical results: in the limit of a highly supply-constrained market (where $\lambda/\tau \to \infty$), the naive $\ensuremath{\mathsf{LR}}$ estimator becomes an unbiased estimator of the $\ensuremath{\mathsf{GTE}}$, while the naive $\ensuremath{\mathsf{CR}}$ estimator is biased.  On the other hand, in the limit of a highly demand-constrained market (where $\lambda/\tau \to 0$), the naive $\ensuremath{\mathsf{CR}}$ estimator becomes an unbiased estimator of the GTE, while the naive $\ensuremath{\mathsf{LR}}$ estimator is biased.  In other words, each of the two naive designs is respectively unbiased in the limits of extreme market imbalance. 
At the same time, we find empirically that neither estimator performs well in the region of moderate market balance. 


Inspired by this finding, we consider $\ensuremath{\mathsf{TSR}}$ and associated estimators that naturally interpolate between the two naive designs depending on market balance. %
Given the findings above, we show that a simple approach to adjusting $a_C$ and $a_L$ as a function of market balance yields performance that balances between the naive $\ensuremath{\mathsf{LR}}$ estimator and the naive $\ensuremath{\mathsf{CR}}$ estimator.  Nevertheless, we show there is significant room for improvement, by adjusting for the types of interference that arise using observations from the $\ensuremath{\mathsf{TSR}}$ experiment.  In particular, we propose a heuristic for a novel interpolating estimator for the $\ensuremath{\mathsf{TSR}}$ design that aims to correct these biases, and yields surprisingly good numerical performance. 



 









\subsection{Theory: Stead-state bias of $\ensuremath{\mathsf{CR}}$ and $\ensuremath{\mathsf{LR}}$ in unbalanced markets}
\label{ssec:ss_theory}


In this subsection, we study theoretically the bias of the steady-state naive $\ensuremath{\mathsf{CR}}$ and $\ensuremath{\mathsf{LR}}$ estimators  in the limits where the market is extremely unbalanced (either demand-constrained or supply-constrained).  The key tool we employ is a characterization of the asymptotic behavior of $Q_{ij}(\infty | a_C, a_L)$ as defined in \eqref{eq:Qijinfty} in the limits where $\lambda/\tau \to 0$ and $\lambda/\tau \to \infty$.  We use this characterization in turn to quantify the asymptotic bias of the naive estimators relative to the $\ensuremath{\mathsf{GTE}}$. We derive these results in the next two subsections and provide a simple example in Section \ref{ssec:Qij_homogeneous} to illustrate the effects.

\subsubsection{Highly demand-constrained markets.}


We start by considering the behavior of naive estimators in the limit where $\lambda/\tau \to 0$.  We start with the following proposition that characterizes behavior of $Q_{ij}(\infty|a_C,a_L)$ as $\lambda/\tau \to 0$.  The proof is in Appendix \ref{app:proofs}.
\begin{proposition}
\label{prop:Qij_demand_constrained}
Fix all system parameters except $\lambda$ and $\tau$, and consider a sequence of systems in which $\lambda/\tau \to 0$.  Then along this sequence,
\begin{equation}
\label{eq:Qij_demand_constrained}
\frac{1}{\lambda} Q_{ij}(\infty | a_C, a_L) \to \sum_{\theta} \sum_{\gamma} \phi_{\gamma,i} p_{\gamma,i}( \theta,j | \v{\rho} ).
\end{equation}
\end{proposition}
The expression on the right hand side depends on both $a_C$ and $a_L$ through $\phi_{\gamma,i}$ and $\v{\rho}$ respectively.  In particular, we recall that $\phi_{\gamma,1} = a_C \phi_\gamma$, $\phi_{\gamma,0} = (1-a_C) \phi_\gamma$, and $\rho(\theta,1) = a_L \rho(\theta)$, $\rho(\theta,0) = (1-a_L) \rho(\theta)$.  In our subsequent discussion in this regime, to emphasize the dependence of $\v{\rho}$ on $a_L$ below, we will write $\v{\rho}(a_L) = (\rho(\theta,j|a_L), \theta \in \Theta, j = 0,1)$.  With this definition, we have  $\rho(\theta,1|a_L) = a_L \rho(\theta)$, $\rho(\theta,0|a_L) = (1-a_L) \rho(\theta)$.

This proposition allows us to characterize the bias of both $\ensuremath{\mathsf{CR}}$ and $\ensuremath{\mathsf{LR}}$ estimators in the demand constrained limit. 
Note that Proposition \ref{prop:Qij_demand_constrained} shows that, in this limit, the (scaled) rate of booking behaves {\em as if} the available listings of type $(\theta,j)$ was {\em exactly} $\rho(\theta,j|a_L)$ for every $\theta$ and treatment condition $j = 0,1$.  That is, it is as if every arriving customer sees the entire mass of listings as being available, and so bookings are immediately replenished. This observation drives our first main result, that in the demand constrained limit the $\ensuremath{\mathsf{CR}}$ estimator is unbiased and $\ensuremath{\mathsf{LR}}$ estimator is biased.



\begin{theorem}
\label{thm:demand_constrained}
Consider a sequence of systems where $\lambda/\tau \to 0$.  Then for all $a_C$ such that $0 < a_C < 1$, $\widehat{\ensuremath{\mathsf{GTE}}}^{\ensuremath{\mathsf{CR}}}(\infty|a_C)/\lambda - \ensuremath{\mathsf{GTE}}/\lambda \to 0$.  However, for $0 < a_L < 1$, generically over parameter values\footnote{Here "generically" means for all parameter values, except possibly for a set of parameter values of Lebesgue measure zero.} we have
$\lim\ \widehat{\ensuremath{\mathsf{GTE}}}^{\ensuremath{\mathsf{LR}}}(\infty|a_C)/\lambda - \ensuremath{\mathsf{GTE}}/\lambda \neq 0$.
\end{theorem}

The full proof can be found in Appendix \ref{app:proofs}. The key insight is that as the market becomes more demand constrained, there is a weakening of the competition between arriving customers, which leads to less interference in a $\ensuremath{\mathsf{CR}}$ experiment. In the limit, the $\ensuremath{\mathsf{CR}}$ estimator becomes unbiased. On the other hand, in an $\ensuremath{\mathsf{LR}}$ experiment there is a positive mass of control \textit{and} treatment listings available in steady state, leading to competition between listings and bias in the $\ensuremath{\mathsf{LR}}$ estimator.














\subsubsection{Heavily supply-constrained markets.}

We now characterize the behavior of naive estimators in the limit where $\lambda/\tau \to \infty$.  We start with the next proposition, where we study the behavior of $Q_{ij}$ as $\lambda/\tau \to \infty$.  The proof is in Appendix \ref{app:proofs}.  To state the proposition, we define:
\[ g_{\gamma,i}(\theta ,j) = \frac{\alpha_{\gamma,i}(\theta,j) v_{\gamma,i}(\theta, j)}{\epsilon_{\gamma,i}}. \]


\begin{proposition}
\label{prop:Qij_supp_constrained}
Fix all system parameters except $\lambda$ and $\tau$, and consider a sequence of systems in which $\lambda/\tau \to \infty$.  Along this sequence, the following limit holds:
\begin{equation}
\label{eq:Qij_supp_constrained}
\frac{1}{\tau}Q_{ij}(\infty | a_C, a_L) \to \sum_\theta \left(\frac{\sum_\gamma \phi_{\gamma,i} g_{\gamma,i}(\theta,j)}{\sum_{i' = 0,1} \sum_{\gamma}\phi_{\gamma,i'} g_{\gamma,i'}(\theta,j)}\right) \rho(\theta,j) \nu(\theta).
\end{equation}
\end{proposition}
As before, the expression on the right hand side depends on both $a_C$ and $a_L$ through $\phi_{\gamma,i}$ and $\v{\rho}$ respectively.  In particular, we recall that $\phi_{\gamma,1} = a_C\phi_\gamma$, $\phi_{\gamma,0} = (1-a_C) \phi_\gamma$, and $\rho(\theta,1) = a_L \rho(\theta)$, $\rho(\theta,0) = (1-a_L)\rho(\theta)$.

A key intermediate result we employ is to demonstrate that in the steady-state in this limit, $s^*(\theta, j|a_C, a_L) \to 0$ for all $\theta,j$.  We know that in the steady state of the mean field limit, the rate at which occupied listings become available must match the rate at which available listings become occupied (flow conservation).  We use this fact to show that to first order in $\lambda/\tau$, in the limit where $\lambda/\tau \to \infty$ we have:
\[ s^*(\theta, j | a_C, a_L) \approx \frac{1}{\lambda/\tau} \cdot \frac{\rho(\theta, j) \nu(\theta)}{\sum_{\gamma} \sum_{i = 0,1} \phi_{\gamma,i} g_{\gamma,i}(\theta, j)}. \]
The proposition follows by using this limit to characterize the choice probabilities.


The proof of the preceding proposition reveals that in the limit where $\lambda/\tau \to \infty$, we have:
\[ p_{\gamma,i}(\theta,j | \v{s}^*(a_C, a_L)) \approx g_{\gamma,i}(\theta,j) s^*(\theta,j|a_C,a_L) = \frac{ \alpha_{\gamma,i}(\theta,j) v_{\gamma,i}(\theta,j) s^*(\theta,j|a_C, a_L)}{ \epsilon_{\gamma,i}}. \]
This preceding expression is the formalization of our intuition that, in the limit where the market is heavily supply-constrained, it is as if each arriving customer seeing an available listing compares only that listing to the outside option; there is no longer any competition {\em between} listings.

We can use the preceding proposition to understand the behavior of the $\ensuremath{\mathsf{GTE}}$, the naive $\ensuremath{\mathsf{LR}}$ estimator, and the naive $\ensuremath{\mathsf{CR}}$ estimator in steady-state, as $\lambda/\tau \to \infty$.  For simplicity, we hold $\tau$ constant and take the limit $\lambda \to \infty$.  In this case, the preceding proposition shows that:
\[ Q_{11}(\infty | 1, 1) \to  \tau \sum_\theta  \rho(\theta) \nu(\theta) ; \ \  Q_{00}(\infty | 0, 0) \to  \tau \sum_\theta  \rho(\theta) \nu(\theta). \]
The global treatment effect $\ensuremath{\mathsf{GTE}} \to 0$ in this limit. 
Bookings occur essentially instantaneously after a listing becomes available, which happens at rate $\tau \sum_\theta  \rho(\theta) \nu(\theta)$. 

We also note that:
\[ Q_{11}(\infty | 1,a_L) \to a_L \tau \sum_{\theta} \rho(\theta) \nu(\theta); \ \ Q_{10}(\infty | 1,a_L) \to (1-a_L) \tau \sum_{\theta} \rho(\theta) \nu(\theta).\]
The preceding two expressions reveal that the steady-state naive $\ensuremath{\mathsf{LR}}$ estimator $\widehat{\ensuremath{\mathsf{GTE}}}^{\ensuremath{\mathsf{LR}}}(\infty|a_L)$ in this setting approaches zero, matching the $\ensuremath{\mathsf{GTE}}$; thus it is asymptotically unbiased.  

It is also now straightforward to see why the $\ensuremath{\mathsf{CR}}$ design will be biased.  Note that:
\[ Q_{11}(\infty | a_C,1) \to a_C \tau \sum_\theta \left(\frac{\sum_\gamma \phi_{\gamma} g_{\gamma,1}(\theta,1)}{\sum_{\gamma}\sum_{i' = 0,1} \phi_{\gamma,i'} g_{\gamma,i'}(\theta,1)}\right) \rho(\theta) \nu(\theta). \] 
An analogous expression holds for $Q_{01}(\infty | a_C, 1)$.  We see that the right hand side reflects the dynamic interference created between treatment and control customers: just as in our simple example, whether or not an available listing is seen by, e.g., a control customer depends on whether it has previously been booked by a treatment customer. That is, customers compete for listings.
As in the example, the naive $\ensuremath{\mathsf{CR}}$ estimator will remain nonzero in general in the limit, even though the $\ensuremath{\mathsf{GTE}}$ approaches zero.

We summarize our discussion in the following theorem.
\begin{theorem}
\label{thm:supp_constrained}
Consider a sequence of systems where $\lambda/\tau \to \infty$.  Then $\ensuremath{\mathsf{GTE}}/\tau \to 0$, and for all $a_L$ such that $0 < a_L < 1$, there also holds $\widehat{\ensuremath{\mathsf{GTE}}}^{\ensuremath{\mathsf{LR}}}(\infty | a_L)/\tau \to 0$.  However, for $0 < a_C < 1$, generically over parameter values we have
$\lim\ \widehat{\ensuremath{\mathsf{GTE}}}^{\ensuremath{\mathsf{CR}}}(\infty|a_C)/\tau - \ensuremath{\mathsf{GTE}}/\tau \neq 0$.
\end{theorem}

Although the preceding theorem shows that the {\em absolute} bias of the naive $\ensuremath{\mathsf{LR}}$ estimator approaches zero, in fact in general the {\em relative} bias $(\widehat{GTE}^{\ensuremath{\mathsf{LR}}}(\infty|a_L) - \ensuremath{\mathsf{GTE}})/\ensuremath{\mathsf{GTE}}$ will not generally approach zero; this is because the $\ensuremath{\mathsf{GTE}}$ is also approaching zero, and so the second-order behavior of the naive $\ensuremath{\mathsf{LR}}$ estimator matters.  This is in contrast to the behavior of the naive $\ensuremath{\mathsf{CR}}$ estimator in the demand-constrained limit: in that limit, the $\ensuremath{\mathsf{GTE}}$ remains nonzero in general, and so the naive $\ensuremath{\mathsf{CR}}$ estimator is both absolutely and relatively unbiased.  Nevertheless, note that relative bias of the naive $\ensuremath{\mathsf{LR}}$ estimator will be significantly smaller than the relative bias of the naive $\ensuremath{\mathsf{CR}}$ estimator in the supply-constrained limit, since the naive $\ensuremath{\mathsf{CR}}$ estimator has a nonzero absolute bias in this limit while the $\ensuremath{\mathsf{GTE}}$ approaches zero.

We finish this subsection by noting Theorems \ref{thm:demand_constrained} and \ref{thm:supp_constrained} are driven by fundamental competition dynamics in the respective limiting regimes, and therefore, we believe they hold under a much more general class of models than the ones considered here. We leave this for future investigation.  



\subsubsection{An example: Homogeneous customers and listings.}
\label{ssec:Qij_homogeneous}

To more clearly understand the behavior of the bias, in this section we apply Propositions \ref{prop:Qij_demand_constrained} and \ref{prop:Qij_supp_constrained} to a simpler setting where 
both listings and customers are homogeneous, i.e., there is only one type of customer and one type of listing.  This example illustrates the symmetry between the two sides of the market and the resulting implications for bias in marketplace experiments. 

Let $v$ denote the control utility 
and $\tilde{v}$ the treatment utility of a customer for a listing. Let $\epsilon$ denote the outside option value of both control and treatment customers,  $\alpha_0(0) = \alpha_1(1) = 1$, and  $\nu(0) = \nu(1) = 1$. %
In this example, we consider two limits: one where $\lambda$ is fixed and $\tau \to \infty$ (the demand-constrained regime), and one where $\tau$ is fixed and $\lambda \to \infty$ (the supply-constrained regime).  

In the first case, when $\tau \to \infty$ with $\lambda$ fixed, if we apply Proposition \ref{prop:Qij_demand_constrained}, we obtain:
\begin{equation}
\label{eq:Qij_homogeneous_demand_const}
\begin{aligned}[c]
Q_{00}(\infty | a_C, a_L) \to & \lambda \cdot \frac{(1-a_C)(1-a_L) \rho v}{\epsilon + \rho v}; %
\\
Q_{10}(\infty | a_C, a_L) \to &\lambda \cdot \frac{a_C(1-a_L)\rho v}{\epsilon + (1-a_L)\rho v + a_L \rho \tilde{v}};%
\\
\end{aligned}
\qquad
\begin{aligned}[c]
Q_{01}(\infty | a_C, a_L) \to &\lambda \cdot \frac{(1-a_C)a_L \rho v}{\epsilon + \rho v};%
\\
Q_{11}(\infty | a_C, a_L) \to & \lambda \cdot \frac{a_C a_L \rho \tilde{v}}{\epsilon + (1-a_L)\rho v + a_L \rho \tilde{v}}.%
\end{aligned}
\end{equation}
In this limit,
\[ \ensuremath{\mathsf{GTE}} \to \lambda \cdot \left(\frac{\rho \tilde{v}}{\epsilon + \rho \tilde{v}} - \frac{\rho v}{\epsilon + \rho v}\right). \]
From these expressions it is clear that the naive $\ensuremath{\mathsf{CR}}$ estimator is unbiased, while the naive $\ensuremath{\mathsf{LR}}$ estimator is biased.  Further, the expressions reveal that listing-side randomization creates interference across listings.

In the second case, when $\lambda \to \infty$ with $\tau$ fixed, if we apply Proposition \ref{prop:Qij_supp_constrained}, we obtain:
\begin{equation}
\label{eq:Qij_homogeneous_supp_const}
\begin{aligned}[c]
    Q_{00}(\infty | a_C, a_L) \to &\tau (1-a_C)(1-a_L) \rho; %
    \\
    Q_{10}(\infty | a_C, a_L) \to &\tau a_C(1-a_L)\rho;%
\end{aligned}
\qquad
\begin{aligned}[c]
    Q_{01}(\infty | a_C, a_L) \to &\tau \cdot \frac{(1-a_C)v}{(1-a_C)v + a_C \tilde{v}} a_L \rho; %
    \\
    Q_{11}(\infty | a_C, a_L) \to &\tau \cdot \frac{a_C\tilde{v}}{(1-a_C)v + a_C \tilde{v}} a_L \rho.%
\end{aligned}
\end{equation}
In this limit,  $\ensuremath{\mathsf{GTE}} \to 0$.  From these expressions it is clear that the naive $\ensuremath{\mathsf{CR}}$ estimator is biased, while the naive $\ensuremath{\mathsf{LR}}$ estimator is unbiased.  Further, these expressions also reveal that customer-side randomization creates interference across customers.

Interestingly, these expressions highlight a remarkable symmetry.  As expected, in the limit of a highly demand-constrained market, customers choose among listings; thus there is competition for customers among listings, and this is the source of potential interference in $\ensuremath{\mathsf{LR}}$ designs.  The expressions reveal that in the limit of a highly supply-constrained market, it is {\em as if} listings choose among customers; thus there is competition among customers, and this is the source of potential interference in $\ensuremath{\mathsf{CR}}$ designs.  Indeed, the expressions for $Q_{01}$ and $Q_{11}$ in  \eqref{eq:Qij_homogeneous_supp_const} take the form of a multinomial logit choice model of listings for customers.  We believe this type of symmetry provides important insight into the nature of experimental design in two-sided markets, and in particular the roots of the interference typically observed in such settings.


\subsubsection{Sign of the bias in $\ensuremath{\mathsf{CR}}$ and $\ensuremath{\mathsf{LR}}$ estimates.}
Theorems \ref{thm:demand_constrained} and \ref{thm:supp_constrained} state that the $\ensuremath{\mathsf{LR}}$ estimate is biased in the demand constrained limit and the $\ensuremath{\mathsf{CR}}$ estimate is biased in the supply constrained limit, but make no claim as to whether the estimators overestimate or underestimate the $\ensuremath{\mathsf{GTE}}$. In general, we cannot provide guarantees for the sign of the bias, as it depends on the distribution of listings, the rates at which listing replenish, and the lift on the individual $\alpha_\gamma(\theta)$ and $v_\gamma(\theta)$ induced by the interventions. However, for a broad class of interventions, we can show that the $\ensuremath{\mathsf{LR}}$ estimate overestimates in the demand constrained limit and $\ensuremath{\mathsf{CR}}$ overestimates in the supply constrained limit. In such cases where we know the bias to be positive, $\ensuremath{\mathsf{CR}}$ and $\ensuremath{\mathsf{LR}}$ experiments can be used to bound the size of the $\ensuremath{\mathsf{GTE}}$. 

We call an intervention \textit{positive} if $\tilde{\alpha}_\gamma(\theta)\tilde{v}_\gamma(\theta) > \alpha_\gamma(\theta)v_\gamma(\theta)$ for all $\gamma$ and $\theta$. Such an intervention can be viewed as an improvement on the platform for all customer and listing type pairs, since for each pair at least one of the customer's consideration probability or utility for the listing type must increase. Note that this class of interventions is broad enough to allow for heterogeneous treatment effects across different $(\gamma, \theta)$ pairs.\footnote{A symmetric analysis can be applied for ``negative" interventions, where $\tilde{\alpha}_\gamma(\theta)\tilde{v}_\gamma(\theta) < \alpha_\gamma(\theta)v_\gamma(\theta)$ for all $\gamma$ and $\theta$; though, of course, interventions known to be negative in advance are less likely to be desirable from the platform's perspective.}

For positive interventions, straightforward applications of Propositions \ref{prop:Qij_demand_constrained} and \ref{prop:Qij_supp_constrained} show that $\widehat{\ensuremath{\mathsf{GTE}}}^{\ensuremath{\mathsf{LR}}}$ overestimates the $\ensuremath{\mathsf{GTE}}$ in the demand constraint limit and $\widehat{\ensuremath{\mathsf{GTE}}}^\ensuremath{\mathsf{CR}}$ overestimates the $\ensuremath{\mathsf{GTE}}$ in the supply constrained limit. The result follows from the fact that in a customer-side (resp., listing-side) experiment in a supply constrained (resp., demand-constrained) setting, the individuals in the treatment group face less competition than they would in the global treatment setting, whereas the individuals in the control group face more competition than in the global control setting. 

\begin{proposition}
\label{prop:cr_lr_positive_bias}
Suppose that the treatment is positive, i.e., $\tilde{\alpha}_\gamma(\theta)\tilde{v}_\gamma(\theta) > \alpha_\gamma(\theta)v_\gamma(\theta)$ for all $\gamma, \theta$.
Then we have the following. 
\begin{enumerate}
    \item $\ensuremath{\mathsf{LR}}$ bias when demand constrained: Consider a sequence of systems where $\lambda/\tau \to 0$. For any $0 < a_L < 1$, we have
    $\lim\ \widehat{\ensuremath{\mathsf{GTE}}}^{\ensuremath{\mathsf{LR}}}(\infty|a_C)/\lambda - \ensuremath{\mathsf{GTE}}/\lambda > 0$. 
    \item $\ensuremath{\mathsf{CR}}$ bias when supply constrained: Consider a sequence of systems where $\lambda/\tau \to \infty$. For any $0 < a_C < 1$,  we have $\lim\ \widehat{\ensuremath{\mathsf{GTE}}}^{\ensuremath{\mathsf{CR}}}(\infty|a_C)/\tau - \ensuremath{\mathsf{GTE}}/\tau > 0$.
\end{enumerate}
\end{proposition}
Further, we find through simulations that $\ensuremath{\mathsf{CR}}$ and $\ensuremath{\mathsf{LR}}$ overestimate the $\ensuremath{\mathsf{GTE}}$ with positive treatments in intermediate ranges of market balance, for all parameter regimes that we study in the examples in this section (see Figure \ref{fig:simulations_hom} as well as  Appendix \ref{app:simulations}). We do find in some cases that the $\ensuremath{\mathsf{TSRI}}$ estimators underestimate the $\ensuremath{\mathsf{GTE}}$, and so, since we plot bias on a log scale, we report the absolute value of the bias.  %






\subsection{Discussion: Violation of SUTVA}

Our results on the bias of the naive $\ensuremath{\mathsf{CR}}$ and $\ensuremath{\mathsf{LR}}$ experiments
can be interpreted through the lens of the classical potential outcomes model. An important result from this literature is that when the {\em stable unit treatment value assumption} (SUTVA) holds, then naive estimators of the sort we consider will be unbiased for the true treatment effect.  SUTVA requires that the treatment condition of units other than a given customer or listing should not influence the potential outcomes of that given customer or listing.  The discussion above illustrates that in the limit where $\lambda \to 0$, there is no interference across customers in the $\ensuremath{\mathsf{CR}}$ design; this is why the naive $\ensuremath{\mathsf{CR}}$ estimator is unbiased.  Similarly, in the limit where $\lambda \to \infty$, there is no interference across listings in the $\ensuremath{\mathsf{LR}}$ design; this is why the naive $\ensuremath{\mathsf{LR}}$ estimator is unbiased.  On the other hand, the cases where each estimator is biased involve interference across experimental units.




\input{TSR}






























\section{Comparison with cluster-randomized experiments}
\label{sec:cluster}

In this section we compare the $\ensuremath{\mathsf{TSR}}$ approach to existing approaches to reduce bias in marketplace experiments.  One such approach is to run a cluster-randomized experiment, which changes the unit of randomization in order to reduce interference effects across units. The typical approach is to divide the marketplace into clusters, such as geographical regions, such that there is less interaction of market participants across different clusters. All participants within a cluster receive the same treatment condition. The platform then estimates the $\ensuremath{\mathsf{GTE}}$ by comparing the outcomes within the treatment clusters versus the outcomes in the control clusters. It is important to note that many markets and social networks are highly connected and it is not possible to avoid all interference across clusters; see, e.g., \cite{holtz2020reducing} for an example in the context of Airbnb. Thus cluster-randomized experiments will reduce but not fully remove the bias.


To compare the performance of the cluster-randomized and $\ensuremath{\mathsf{TSR}}$ approaches, we use our existing model to define a regime that gives the best-case performance for cluster-randomized experiments, where there are tightly clustered preferences in the marketplace and the platform knows ex-ante the true clusters (without having to learn them).

The simulations suggest that cluster-randomized estimators offer substantial bias reductions over when the market is tightly clustered, but these improvements diminish if the market becomes more interconnected. The $\ensuremath{\mathsf{TSR}}$ estimators, however, offer bias reductions in both clustered and interconnected markets and, in interconnected markets, are less biased than the cluster-randomized estimator. 
In our simulations, the variance of the cluster-randomized estimator is lower than that of $\ensuremath{\mathsf{TSRI\text{-}2}}$ in our example, although the variance of the cluster-randomized estimator will likely change if we deviate from this best-case scenario with perfect knowledge of the clusters, identical listings within clusters, and uniform treatment effects across different clusters.  Hence, while our model for market clusters is stylized, we believe our results suggest that $\ensuremath{\mathsf{TSR}}$ designs can be a useful alternative to cluster-randomized experiments in interconnected markets. 





\subsection{Market setup for cluster randomization}

We consider the case where clusters are defined on the listing side, so that the cluster-randomized experiment is expected to improve upon a listing-randomized experiment.\footnote{Because we analyze a model where customers are short-lived while listings remain on the platform, it is likely that the platform has more information on the listings, and is better able to learn clusters on the listing side.} To induce a clustered structure, we model a setting with two customer types and two listing types, where each type of customer prefers a different type of listing. Formally, there are customer types $\{\gamma_1, \gamma_2\}$ and listing types $\{\theta_1, \theta_2\}$. All customers consider all listings ($\alpha_\gamma(\theta)=1$ for all $\gamma, \theta$) but customers have different utilities for different listings.\footnote{Alternatively, we can induce a clustered structure by modifying the consideration probabilities $\alpha_\gamma(\theta)$. Both approaches of modifying the $\alpha_\gamma(\theta)$ and modifying the $v_\gamma(\theta)$ are equivalent in the mean field model.} The global control utilities $v_\gamma(\theta)$ have the following form:
\medskip

\begin{center}
\begin{tabular}{ c | c c } 
 & $\theta_1$ & $\theta_2$ \\
\hline
$\gamma_1$ & $x$ & $y$ \\
$\gamma_2$ & $y$ & $x$ \\
\end{tabular}
\end{center}
\smallskip
where $x\geq y \geq 0$. 
Note that if $y=0$, then the market can be perfectly decomposed into two sub-markets, where customers of type $\gamma_1$ (resp., $\gamma_2$) \textit{only} book listings of type $\theta_1$ (resp., $\theta_2$). If $y=x$, then each customer prefers both listings equally. Thus we can interpret the ratio $y/x$ as a measure of how equally a customer prefers both products, where intuitively the market is tightly clustered when $y/x$ is small. We call $y/x$ the \textit{preference ratio}.






The platform then runs a cluster randomized experiment where it first assigns listings to clusters and then randomizes entire clusters to either treatment or control. 
In practice, the platform must \textit{learn} how to create the clusters, likely through observational data in the global control setting,\footnote{See \cite{holtz2020reducing} in the context of marketplaces and \cite{Ugander13} in the context of social networks.} but in these simulations, we assume that the platform observes the cluster structure perfectly. The platform assigns all $\theta_1$ listings to one cluster and $\theta_2$ listings to another, and runs a completely randomized design on the clusters, assigning one of the clusters to treatment and one to control. For simplicity, assume that the intervention has a multiplicative lift $\delta>1$ on all customer-listing pairs, so that the treatment utilities satisfy $\tilde{v}_\gamma(\theta) = \delta v_\gamma(\theta)$. %

The cluster-randomized estimator $\widehat{\ensuremath{\mathsf{GTE}}}^{\ensuremath{\mathsf{Cluster}}}$, with clusters defined on the listing side, compares the (scaled) rate of bookings of listings in treatment clusters to the rate of bookings of listings in control clusters.
Formally, in the mean field setting, once the clusters are randomized, let $Z$ denote the mass of listings assigned to a treatment cluster. %
Then
\begin{equation}
\label{eq:naive_cluster}
 \widehat{\ensuremath{\mathsf{GTE}}}^{\ensuremath{\mathsf{Cluster}}}(T | Z) = \frac{Q_{11}(T | 1,Z)}{Z} - \frac{Q_{10}(T | 1,Z)}{1-Z}. 
\end{equation}
We can similarly define an analogous estimator in the finite model.



The full set of parameters is as follows. The market has an equal number of listings of both types, so that $m^{(N)}(\theta_1) = m^{(N)}(\theta_2) = 0.5$, and the same arrival rate for both customers types ($\lambda_{\gamma_1}^{(N)} = \lambda_{\gamma_2}^{(N)}$). We set $x=0.5$ and vary $y \in [0,1]$. We set $\delta=1.3$. We fix $\tau=1$ and  consider $\lambda \in \{0.1, 1, 10\}$. 

\subsection{Results}

Figure \ref{fig:cluster_vary_pref_ratio} shows how the performance of the estimators change when we vary the preference ratio (at a fixed market balance). We choose $\ensuremath{\mathsf{TSRI\text{-}2}}$ as a representative estimator for the $\ensuremath{\mathsf{TSR}}$ approach; see Figure \ref{fig:cluster_vary_pref_ratio_all_estimators} in Appendix \ref{app:cluster} for a comparison with all estimators. 

We see a clear takeaway that, perhaps unsurprisingly, the cluster randomized estimator offer substantial bias improvements when the market is tightly clustered (i.e., a customer strongly prefers one type of item over another) but offer little reduction in bias when the market is more interconnected.  In particular, the cluster-randomized estimator outperforms the $\ensuremath{\mathsf{TSR}}$ estimators when the market is tightly clustered, while the $\ensuremath{\mathsf{TSR}}$ estimators outperform the cluster-randomized estimator when the market is more connected. 


The standard error changes little across the preference ratios, with the standard error of the cluster-randomized estimate lower than that of $\ensuremath{\mathsf{TSRI\text{-}2}}$, although the variance of the cluster-randomized estimator will likely change if we deviate from this best-case scenario with perfect knowledge of the clusters, identical listings within clusters, and uniform treatment effects across clusters.  In our scenario, the cluster-randomized estimator has lower $\mathrm{RMSE}$ for tightly clustered markets and higher otherwise. 



\begin{figure}
    \centering
    \begin{tabular}{l l }
        \includegraphics[height=.28\textwidth]{figs_sims/cluster/sims_bias_varying_pref_ratio_lam_1_error_bars_mod}
        &
        \includegraphics[height=.28\textwidth]{figs_sims/cluster/sims_bias_over_LR_varying_pref_ratio_lam_1_error_bars_mod.png}
        \\
        \includegraphics[height=.28\textwidth]{figs_sims/cluster/sims_se_varying_pref_ratio_lam_1_error_bars_mod.png}
        &
        \includegraphics[height=.28\textwidth]{figs_sims/cluster/sims_rmse_varying_pref_ratio_lam_1_error_bars_mod}
    \end{tabular}
    \caption{Simulations of cluster-randomized estimator and TSR estimators as preference ratio $y/x$ varies. Relative demand is fixed at $\lambda/\tau=1$. We fix $x=0.5$ and vary $y$. Bootstrapped 95 percentile confidence intervals are provided for each statistic.
    }
    \label{fig:cluster_vary_pref_ratio}
\end{figure}












\section{Conclusion}
\label{sec:conclusion}

This paper proposes a general mean field framework to study the dynamics of inventory bookings in two-sided platforms, and we leverage this framework to study the design and analysis of a number of different experimental designs and estimators.  We study both commonly used designs and estimators ($\ensuremath{\mathsf{CR}}$, $\ensuremath{\mathsf{LR}}$), and also introduce a family of more general two-sided randomization designs and estimators ($\ensuremath{\mathsf{TSR}}$).  Our work sheds light on the market conditions in which each approach to estimation performs best, based on the relative supply and demand balance in the marketplace.

For bias minimization, we suggest two directions for future work. The first is further optimization of the $\ensuremath{\mathsf{TSR}}$ design as a standalone experiment design. We have proposed three natural $\ensuremath{\mathsf{TSR}}$ estimators, but the space of both designs and estimators is much richer and it is worth asking which are optimal with respect to bias and variance, as well as how this answer may change with differing market conditions. The second direction is to develop the $\ensuremath{\mathsf{TSR}}$ design as a method to debias one-sided experiments. The design allows us to measure competition effects between customers and between listings; this observation suggests that these measurements can be used to approximately debias existing $\ensuremath{\mathsf{CR}}$ and $\ensuremath{\mathsf{LR}}$ experiments, providing another route for platforms to utilize two-sided randomization designs.


To make technical progress in this paper, we employed several simplifying assumptions on the choice model and booking behavior. %
We believe that our core insight of market balance mediating competition effects, and thus affecting the resulting bias in an experiment, extends to other settings as well. 
We hypothesize, however, that some of our results are more robust to modeling choices than others. For example, the result that the $\ensuremath{\mathsf{CR}}$ estimator is unbiased in the demand constrained limit depends only on the fact that booked listings are replenished in between customer arrivals, and likely extends to any reasonable choice model. On the other hand, the result that the absolute bias of the $\ensuremath{\mathsf{LR}}$ estimator approaches 0 in the supply constrained limit may be more sensitive to different choice models, and in particular how the customer weighs options on the platform compared to the outside option. In this supply constrained limit, we conjecture that the relative performance of the two experiment types still holds even in modified settings; that is, the $\ensuremath{\mathsf{LR}}$ estimator has lower bias than the $\ensuremath{\mathsf{CR}}$ bias. Further, we believe that the approach of using $\ensuremath{\mathsf{TSR}}$ to observe competition effects and heuristically debias estimators also extends beyond our model.



Another practical consideration for platforms is that not all experiment designs are suitable for all types of interventions. In this paper, we have largely focused on interventions such as user interface changes that change how an \textit{individual} customer perceives an \textit{individual} listing. There are also other interventions that operate between \textit{subsets} of customers or \textit{subsets} of listings. For example, a modification in the ranking algorithm over the listings changes operates not on an individual customer-listing pair, but rather changes how a customer interacts with a subset of listings. This intervention is more conducive to a $\ensuremath{\mathsf{CR}}$ experiment than an $\ensuremath{\mathsf{LR}}$ or $\ensuremath{\mathsf{TSR}}$ experiment. Beyond these feasibility constraints, it remains an open question whether certain types of experiments lead to lower bias in different classes of interventions. 


Finally, we emphasize the importance of inference in these settings, which we do not study in this paper. In practice, standard errors are also estimated ``naively": they are typically computed assuming independence of observations. However, because of interference, observations are clearly not independent. %
In these settings, how biased might the standard error estimates be? How can experimenters derive valid confidence intervals in these settings? Such questions are critical for any platforms controlling the false positive and false negative results arising from their experiments. 



\section{Experiments: Designs and estimators}
\label{sec:experiments}


In this section, we leverage the framework developed in the previous section to undertake a study of experimental designs a platform might employ to test interventions in the marketplace.  For simplicity, we focus on interventions that change the {\em choice probability} of one or more types of customers for one or more types of listings, and we assume the platform is interested in estimating the resulting rate at which bookings take place.  However, we believe the same approach we employ here can be applied to study other types of interventions and platform objectives as well.  %

Formally, the platform's goal is to design experiments with associated estimators to assess the performance of the intervention (the {\em treatment}), relative to the status quo (the {\em control}).  In particular, the platform is interested in determining the steady-state rate of booking when the entire market is in the treatment condition (i.e., {\em global treatment}), compared to the steady-state rate of booking when the entire market is in the control condition (i.e., {\em global control}).  We refer to the difference of these two rates as the {\em global treatment effect} ($\ensuremath{\mathsf{GTE}}$). We focus on these steady-state quantities as a platform is typically interested in the long-run effect of an intervention. 

Two types of canonical experimental designs are typically employed in practice: {\em listing-side randomization} (denoted $\ensuremath{\mathsf{LR}}$) and {\em customer-side randomization} (denoted $\ensuremath{\mathsf{CR}}$).  In the former design, listings are randomized to treatment or control; in the latter design, customers are randomized to treatment or control.  Each design also has an associated natural ``naive" estimator of booking rates, that is, a (scaled) difference in means estimators for the two groups.  As we discuss, these estimators will typically be biased, due to interference effects.

The $\ensuremath{\mathsf{LR}}$ and $\ensuremath{\mathsf{CR}}$ designs are special cases of a novel, more general {\em two-sided randomization} ($\ensuremath{\mathsf{TSR}}$) design that we introduce in this work, where {\em both} listings and customers are randomized to treatment and control simultaneously. As we discuss, this type of experiment can be combined with design and analysis techniques to reduce bias. On the design side, $\ensuremath{\mathsf{TSR}}$ designs allow us to construct experiments that interpolate between $\ensuremath{\mathsf{LR}}$ and $\ensuremath{\mathsf{CR}}$ designs in such a way that bias is reduced. On the analysis side,  $\ensuremath{\mathsf{TSR}}$ designs allow us to observe different competition effects, that we can use to heuristically debias our estimators. ($\ensuremath{\mathsf{TSR}}$ designs were also independently introduced and studied in recent work by \cite{bajari2019double}; see Section \ref{sec:related} for discussion.)  In the next subsection we develop the relevant formalism for these designs; we then subsequently define natural ``naive" estimators that are commonly used for the $\ensuremath{\mathsf{LR}}$ and $\ensuremath{\mathsf{CR}}$ designs, as well as an estimator for a $\ensuremath{\mathsf{TSR}}$ design.  In the remainder of the paper we study the bias of these different designs and estimators under different market conditions.






\subsection{Experimental design}
Since $\ensuremath{\mathsf{CR}}$ and $\ensuremath{\mathsf{LR}}$ are special cases of a $\ensuremath{\mathsf{TSR}}$ design, we first describe how to embed $\ensuremath{\mathsf{TSR}}$ experimental designs into our model, and then subsequently describe $\ensuremath{\mathsf{CR}}$ and $\ensuremath{\mathsf{LR}}$ designs in our model.

\textbf{Treatment condition.}  We consider a binary treatment: every customer and listing in the market will either be in {\em treatment} or {\em control}.   (Generalization of our model to more than two treatment conditions is relatively straightforward.)  We model the treatment condition by expanding the set of customer and listing types.  For every customer type $\gamma \in \Gamma$, we create two new customer types $(\gamma, 0), (\gamma, 1)$; and for every listing type $\theta \in \Theta$, we create a two new listing types $(\theta, 0), (\theta, 1)$.  The types $(\gamma, 0), (\theta, 0)$ are {\em control types}; the types $(\gamma, 1), (\theta, 1)$ are {\em treatment types}.

\textbf{Two-sided randomization.} In this design, randomization takes place on both sides of the market simultaneously. We assume that a fraction $a_C$ of customers are randomized to treatment, and a fraction $1 -a_C$ to control, independently; and we assume that a fraction $a_L$ of listings are randomized to treatment, and a fraction $1 - a_L$ to control, independently.  %

\textbf{Treatment as a choice probability shift.} Examples of interventions that platforms may wish to test include the introduction of higher quality photos for a hotel listing on a lodging site, showing previous job completion rates of a freelancer on an online labor market, or reducing the friction for an item in the checkout flow. 
Such interventions change the choice probability of listings by customers either through the consideration probabilities or perceived utility for a listing.  In particular, we continue to assume the multinomial logit choice model, and we assume that for a type $\gamma$ customer and a type $\theta$ listing that have been given the intervention, the utility becomes $\tilde{v}_\gamma(\theta) > 0$
and the probability of inclusion in the consideration set becomes $\tilde{\alpha}_\gamma(\theta) > 0$.  Since we focus on changes in choice probabilities, we assume that the holding time parameter of a listing of type $\theta$ is $\nu(\theta)$, regardless of whether it is assigned to treatment or control.\footnote{Our current work allows us to relatively easily incorporate $\nu$ depending on treatment condition of the listing, and as such we can extend our results to study $\ensuremath{\mathsf{LR}}$ designs where $\nu$ varies with treatment condition.  In general, however, when customers are also randomized to treatment or control, the holding time parameter of a listing should also depend on the treatment condition of the customer who booked that listing.  Adapting our framework to incorporate this possibility remains an interesting direction for future work.}

In the $\ensuremath{\mathsf{TSR}}$ designs that we consider, a key feature is that the intervention is applied only when {\em a treated customer interacts with a treated listing}.  For example, when an online labor marketplace decides to show previous job completion rates of a freelancer as an intervention,
only treated customers can see these rates, and they only see them when they consider treated freelancers.  We model this by redefining quantities in the experiment as follows:
\begin{align}
 v_{\gamma,0}(\theta, 0) = v_{\gamma, 1}(\theta, 0) = v_{\gamma,0}(\theta, 1) = v_\gamma(\theta);&\quad v_{\gamma,1}(\theta, 1) = \tilde{v}_\gamma(\theta);\label{eq:v_expt}\\
 \alpha_{\gamma,0}(\theta, 0) = \alpha_{\gamma, 1}(\theta, 0) = \alpha_{\gamma,0}(\theta, 1) = \alpha_\gamma(\theta);&\quad 
 \alpha_{\gamma,1}(\theta, 1) = \tilde{\alpha}_\gamma(\theta);
  \label{eq:alpha_expt}\\
\epsilon_{\gamma,0} = \epsilon_{\gamma,1} &= \epsilon_\gamma; \label{eq:epsilon_expt}\\
\nu(\theta,0) = \nu(\theta,1) &= \nu(\theta).\label{eq:nu_expt}
\end{align}
This definition is a natural way to incorporate randomization on each side of the market.  However, we remark here that it is not necessarily canonical; for example, an alternate design would be one where the intervention is applied when {\em either} the customer has been treated {\em or} the listing has been treated.  Even more generally, the design might randomize whether the intervention is applied, based on the treatment condition of the customer and the listing.  In all likelihood, the relative advantages of these designs would depend not only on the bias they yield in any resulting estimators, but also in the variance characteristics of those estimators.  We leave further study and comparison of these designs to future work. %

\textbf{Customer-side and listing-side randomization.}  Two canonical special cases of the $\ensuremath{\mathsf{TSR}}$ design are as follows.  When $a_L = 1$, all listings are in the treatment condition; in this case, randomization only takes place on the customer side of the market.  This is the customer-side randomization ($\ensuremath{\mathsf{CR}}$) design.  When $a_C = 1$, all customers are in the treatment condition and randomization only takes place on the listing side of the market.  This is the listing-side randomization ($\ensuremath{\mathsf{LR}}$) design.

\textbf{System dynamics.}  With the specification above, it is straightforward to adapt our mean field system of ODEs, cf.~\eqref{eq:ODE}, and the associated choice model \eqref{eq:meanfieldchoice}, to this setting.  The key changes are as follows:
\begin{enumerate}
\item The mass of control (resp., treatment) listings of type $(\theta, 0)$ (resp., $(\theta, 1)$) becomes $(1-a_L)\rho(\theta)$ (resp., $a_L \rho(\theta)$).  In other words, abusing notation, we define $\rho(\theta, 0) = (1-a_L) \rho(\theta)$, and $\rho(\theta, 1) = a_L\rho(\theta)$.
\item The arrival rate of control (resp., treatment) customers of type $(\gamma, 0)$ (resp., $(\gamma, 1)$) becomes $(1-a_C)\lambda \phi_\gamma$ (resp., $a_C \lambda \phi_\gamma$).  Thus we define $\phi_{\gamma,0} = (1-a_C)\phi_\gamma$, and $\phi_{\gamma,1} = a_C\phi_\gamma$.
\item The choice probabilities are defined as in \eqref{eq:meanfieldchoice}, with the relevant quantities defined in \eqref{eq:v_expt}-\eqref{eq:epsilon_expt}.
\end{enumerate}

Using Proposition \ref{prop:ODE_trajectory} and Theorems \ref{thm:ODE_ss}, we know that there exists a unique solution to the resulting system of ODEs; and that there exists a unique limit point to which all trajectories converge, regardless of initial condition.  This limit point is the steady state for a given experimental design.  For a $\ensuremath{\mathsf{TSR}}$ experiment with treatment customer fraction $a_C$, and treatment listing fraction $a_L$, we use the notation $\v{s}_t(a_C, a_L) = (s_t(\theta, j) | a_C, a_L), \theta \in \Theta, j \in \{0,1\})$ to denote the ODE trajectory, and we use $\v{s}^*(a_C, a_L) = (s^*(\theta, j) | a_C, a_L), \theta \in \Theta, j \in \{0,1\})$ to denote the steady state.

\textbf{Rate of booking}.  In our subsequent development, it will be useful to have a shorthand notation for the rate at which bookings of listings of treatment condition $j \in \{0,1\}$ are made by customers of treatment condition $i \in \{0,1\}$, in the interval $[0,T]$.  In particular, we define:
\begin{equation}
\label{eq:QijT}
Q_{ij}(T | a_C, a_L) = \frac{\lambda}{T} \int_0^T \sum_\theta \sum_\gamma \phi_{\gamma,i} p_{\gamma,i}(\theta, j | \v{s}_t(a_C, a_L))\; dt.
\end{equation}
Since $\v{s}_t(a_C, a_L)$ is globally asymptotically stable, bounded, and converges to $\v{s}^*(a_C, a_L)$ as $t \to \infty$, we have:
\begin{equation}
\label{eq:Qijinfty}
 Q_{ij}(\infty | a_C, a_L) \triangleq \lim_{T \to \infty} Q_{ij}(T | a_C, a_L) = \lambda \sum_\theta \sum_\gamma \phi_{\gamma,i} p_{\gamma,i}(\theta, j | \v{s}^*(a_C, a_L)).
\end{equation}

\textbf{Global treatment effect.}  Recall we assume the {\em steady-state rate of booking} is the quantity of interest to the platform.  In particular, the platform is interested in the change in this rate from the {\em global control} condition ($a_C = 0, a_L = 0$) to the {\em global treatment} condition ($a_C = 1, a_L = 1$).  

In the global control condition, the steady state rate at which customers book is: $Q^{\ensuremath{\mathsf{GC}}} = Q_{00}(\infty | 0, 0)$, and in the global treatment condition, the steady state rate at which customers book is $Q^{\ensuremath{\mathsf{GT}}} = Q_{11}(\infty | 1,1)$.
Under these definitions, the global treatment effect is $\ensuremath{\mathsf{GTE}} = Q^{\ensuremath{\mathsf{GT}}} - Q^{\ensuremath{\mathsf{GC}}}$.  

We remark that the rate of booking decisions made by arriving customers will change over time, even if the market parameters are constant over time (including the arrival rates of different customer types, as well as the utilities that customers have for each listing type).  This transient change in booking rates is driven by changes in the state $\v{s}_t$; in general, such fluctuations will lead the transient rate of booking to differ from the steady-state rate, for all values of $a_C$ and $a_L$ (including global treatment and global control).  
In this work, we focus on the steady state quantities to capture, informally, the long run change in behavior due to an intervention.
\footnote{Note that in two-sided markets, certain types of interventions will also cause long-run {\em economic} equilibration due to strategic responses on the part of market participants; for example, if prices are lowered during an experiment, this may affect entry decisions of both buyers and sellers, and thus the long-run market equilibrium.  While our model allows the choice probabilities to change due to treatment, a more complete analysis of long run economic equilibration due to interventions remains a direction for future work.}

\subsection{Estimators: Transient and steady state}

The goal of the platform is to use the experiment to estimate $\ensuremath{\mathsf{GTE}}$.  In this section we consider estimators the platform might use to estimate this quantity.  We first consider the $\ensuremath{\mathsf{CR}}$ and $\ensuremath{\mathsf{LR}}$ designs, and we define ``naive'' estimators that the platform might use to estimate the global treatment effect.  These designs and estimators are those most commonly used in practice.  We define these estimators during the transient phase of the experiment %
and then define the associated steady-state versions of these estimators.  
Finally, we combine these estimation approaches in a natural heuristic that can be employed for any general $\ensuremath{\mathsf{TSR}}$ design.

{\bf Estimators for the $\ensuremath{\mathsf{CR}}$ design.}
We start by considering the $\ensuremath{\mathsf{CR}}$ design, i.e., where $a_L = 1$ and $a_C \in (0,1)$.  A simple naive estimate of the rate of booking is to measure the rate at which bookings are made in a given interval of time by control customers, and compare this to the analogous rate for treatment customers.  Formally, suppose the platform runs the experiment for the interval  $t \in [0, T]$, with a fraction $a_C$ of customers in treatment.   
The rate at which  customers of treatment condition $i \in \{0,1\}$ book in this period is $Q_{i1}(T | a_C, 1)$.
The {\em naive $\ensuremath{\mathsf{CR}}$ estimator} is the difference between treatment and control rates, where we correct 
for differences in the size of the control and treatment groups, by scaling with the respective masses:%
\begin{equation}
\label{eq:naive_CR}
 \widehat{\ensuremath{\mathsf{GTE}}}^{\ensuremath{\mathsf{CR}}}(T | a_C) = \frac{Q_{11}(T | a_C,1)}{a_C} - \frac{Q_{01}(T | a_C,1)}{1-a_C}. 
 \end{equation}
We let $\widehat{\ensuremath{\mathsf{GTE}}}^{\ensuremath{\mathsf{CR}}}(\infty | a_C) = Q_{11}(\infty | a_C,1)/a_C - Q_{01}(\infty | a_C,1)/(1-a_C)$ denote the steady-state naive $\ensuremath{\mathsf{CR}}$ estimator.

{\bf Estimators for the $\ensuremath{\mathsf{LR}}$ design.}
Analogously, we can define a naive estimator for the $\ensuremath{\mathsf{LR}}$ design, i.e., where $a_C = 1$ and $a_L \in (0,1)$.  Formally, suppose the platform runs the experiment for the interval  $t \in [0, T]$, with fraction $a_L$ of listings in treatment.   
The rate at which listings with treatment condition $j \in \{0,1\}$ are booked in this period is $Q_{1j}(T | 1, a_L)$.  The {\em naive $\ensuremath{\mathsf{LR}}$ estimator} is the difference between treatment and control rates, scaled by the mass of listings in each group:
\begin{equation}
\label{eq:naive_LR}
 \widehat{\ensuremath{\mathsf{GTE}}}^{\ensuremath{\mathsf{LR}}}(T | a_L) = \frac{Q_{11}(T | 1,a_L)}{a_L} - \frac{Q_{10}(T | 1,a_L)}{1-a_L}. 
\end{equation}
We let $\widehat{\ensuremath{\mathsf{GTE}}}^{\ensuremath{\mathsf{LR}}}(\infty | a_L) = Q_{11}(\infty | 1,a_L)/a_L - Q_{10}(\infty | 1,a_L)/(1-a_L)$ denote the corresponding steady-state naive $\ensuremath{\mathsf{LR}}$ estimator.  %


{\bf Estimators for the $\ensuremath{\mathsf{TSR}}$ design.}  As with the $\ensuremath{\mathsf{LR}}$ and $\ensuremath{\mathsf{CR}}$ designs, it is possible to design a natural naive estimator for the $\ensuremath{\mathsf{TSR}}$ design as well.  In particular, we have the following definition of the {\em naive $\ensuremath{\mathsf{TSR}}$ estimator}:
\begin{equation}
\label{eq:naive_TSR_est}
 \widehat{\ensuremath{\mathsf{GTE}}}^{\ensuremath{\mathsf{TSRN}}}(T | a_C, a_L) = \frac{Q_{11}(T | a_C,a_L)}{a_Ca_L} - \frac{Q_{01}(T | a_C,a_L) + Q_{10}(T | a_C, a_L) + Q_{00}(T | a_C, a_L)}{1 - a_C a_L}.
\end{equation}
To interpret this estimator, observe that the first term is the normalized rate at which treatment customers booked treatment listings in the experiment; we normalize this by $a_C a_L$, since a mass $a_C$ of customers are in treatment, and a mass $a_L$ of listings are in treatment.  This first term estimates the global treatment rate of booking.  The sum $Q_{01}(T | a_C,a_L) + Q_{10}(T | a_C, a_L) + Q_{00}(T | a_C, a_L)$ is the total rate at which control bookings took place: either because the customer was in the control group, or because the listing was in the control group, or both.  (Recall that in the $\ensuremath{\mathsf{TSR}}$ design, the intervention is only seen when treatment customers interact with treatment listings.)  This is normalized by the complementary mass, $1 - a_C a_L$.  This second term estimates the global control rate of booking.  As before, we can define a steady-state version of this estimator as $\widehat{\ensuremath{\mathsf{GTE}}}^{\ensuremath{\mathsf{TSRN}}}(\infty | a_C, a_L)$, with the steady-state versions of the respective quantities on the right hand side of \eqref{eq:naive_TSR_est}.

It is straightforward to check that as $a_L \to 1$, we have $\widehat{\ensuremath{\mathsf{GTE}}}^{\ensuremath{\mathsf{TSRN}}}(T | a_C, a_L) \to \widehat{\ensuremath{\mathsf{GTE}}}^{\ensuremath{\mathsf{CR}}}(T | a_C)$, the naive $\ensuremath{\mathsf{CR}}$ estimator.  Similarly, as $a_C \to 1$, we have $\widehat{\ensuremath{\mathsf{GTE}}}^{\ensuremath{\mathsf{TSRN}}}(T | a_C, a_L) \to \widehat{\ensuremath{\mathsf{GTE}}}^{\ensuremath{\mathsf{LR}}}(T | a_L)$, the naive $\ensuremath{\mathsf{LR}}$ estimator.  In this sense, the naive $\ensuremath{\mathsf{TSR}}$ estimator naturally ``interpolates" between the naive $\ensuremath{\mathsf{LR}}$ and $\ensuremath{\mathsf{CR}}$ estimators.  In the next section, we exploit this interpolation to choose $a_C$ and $a_L$ as a function of market conditions in such a way as to reduce bias.  

More generally, the $\ensuremath{\mathsf{TSR}}$ design also contains much more information about competition in the marketplace, and the resulting interference effects, than either the $\ensuremath{\mathsf{CR}}$ or $\ensuremath{\mathsf{LR}}$ designs.  Inspired by this observation, together with the idea of interpolating between the naive $\ensuremath{\mathsf{CR}}$ estimator and the naive $\ensuremath{\mathsf{LR}}$ estimator, in Section \ref{ssec:TSR_est} we explore alternative, more sophisticated $\ensuremath{\mathsf{TSR}}$ estimators that heuristically debias interference due to competition effects.  As we show, these estimators can offer substantial bias reduction over the naive $\ensuremath{\mathsf{TSRN}}$ estimator above.












































\section{Introduction}
\label{sec:intro}


We develop a framework to study experiments (also known as {\em A/B tests}) that two-sided platform operators routinely employ to improve the platform. Platforms use experiments to test many types of interventions that affect interactions between participants in the market; examples include features that change the process by which buyers search for sellers or interventions that alter the information the platform shares with buyers. The goal of the experiment is to introduce the intervention to some fraction of the market and use the resulting outcomes to estimate the effect if the intervention were introduced to the entire market. Platforms rely on these estimated effect sizes to make decisions about whether or not to launch the intervention to the entire market.

However, in marketplace experiments, these estimates are often biased due to interference between market participants. Market participants interact and compete with each other and, as a result, the treatment assigned to one individual may influence the behavior of another individual. These interactions violate the Stable Unit Treatment Value Assumption (SUTVA) (\cite{ImbensRubin15}) that guarantees unbiased estimates of the treatment effect. Previous work has shown that the resulting bias can be quite large, and at times as large as the treatment effect itself (\cite{Blake14, Fradkin2015SearchFA, holtz2020reducing}). In this work, we model the platform competition dynamics, investigate how they influence the performance of different canonical experimental designs, and also introduce novel designs that can yield improved performance.


We are particularly motivated by marketplaces where customers do not purchase goods, but rather {\em book} them for some amount of time.
This covers a broad array of platforms, including freelancing (e.g., Upwork), lodging (e.g., Airbnb and Booking.com), and many services (tutoring, dogwalking, child care, etc.). While we explicitly model such a platform, the model we describe also captures features of a platform where goods are bought and supply must be replenished for future demand.



Our model consists of a fixed number of {\em listings}; {\em customers} arrive sequentially over (continuous) time.   For example, in an online labor platform, a freelancer offering work is a listing, and a client looking to hire a freelancer is a customer. On a lodging site, listings include hotel rooms or private rooms and customers are travelers wanting to book. Naturally, an arriving customer can only book {\em available} listings (i.e., those not currently booked).  The customer forms a consideration set from the set of available listings and then, according to a choice model, chooses which listing to book from this set (including an outside option).  
Once a listing is booked, it is occupied and becomes unavailable until the end of its occupancy time.


We focus on {\em interventions} by the platform that change the parameters governing the choice probability of the customer, such as those described above; we refer to the new choice parameters as the {\em treatment} model and the baseline as the {\em control} model.\footnote{The same modeling framework that we employ in this paper can be used to consider interventions that change other parameters, such as customer arrival rates or the time that listings remain occupied when booked; such application is outside the scope of our current work.}  
We assume the platform wants to use an experiment to assess the difference between the rate at which bookings would occur if all choices were made according to the treatment parameters %
and the corresponding rate if all choices were made according to the control parameters %
.  This difference is the {\em global treatment effect} or $\ensuremath{\mathsf{GTE}}$.  In particular, we assume that the quantity of interest is the {\em steady-state} (or long-run) $\ensuremath{\mathsf{GTE}}$, i.e., the long-run average difference in rental rates.\footnote{Our framework can also be used to evaluate other metrics of interest based on experimental outcomes; for simplicity we focus on rate of booking in this work.}

Most platforms employ one of two simple designs for testing such an intervention: either {\em customer-side randomization} (what we call the $\ensuremath{\mathsf{CR}}$ design) or {\em listing-side randomization} (what we call the $\ensuremath{\mathsf{LR}}$ design).  In the $\ensuremath{\mathsf{CR}}$ design, customers are randomized to treatment or control.  All customers in treatment make choices according to the treatment choice model and all customers in control make choices according to the control choice model.  In the $\ensuremath{\mathsf{LR}}$ design, listings are randomized to treatment or control, and the utility of a listing is then determined by its treatment condition.
As a result, in the $\ensuremath{\mathsf{LR}}$ design, in general each arriving customer will consider {\em some} listings in the treatment condition and {\em some} listings in the control condition.  As an example, suppose the platform decides to test an intervention that shows badges for certain listings.  In the $\ensuremath{\mathsf{CR}}$ design, all treatment customers see the badges and no control customers see the badges.  In the $\ensuremath{\mathsf{LR}}$ design, all customers see the badges on treated listings, and do not see them on control listings.

Each of these designs are associated with natural estimators.  In the $\ensuremath{\mathsf{CR}}$ design, the platform measures the difference in the rate of bookings between treatment and control customers; %
this is what we call the {\em naive $\ensuremath{\mathsf{CR}}$ estimator}.  In the $\ensuremath{\mathsf{LR}}$ design, the platform measures the difference in the rate at which treatment and control listings are booked;
this is what we call the {\em naive $\ensuremath{\mathsf{LR}}$ estimator}.

To develop some intuition for the potential biases, 
first consider an idealized {\em static} setting where listings are instantly replenished upon being booked; in other words, every arriving customer sees the  full set of original listings as available.  
As a result, in the $\ensuremath{\mathsf{CR}}$ design there is no  interference between treatment and control customers, and consequently the $\ensuremath{\mathsf{CR}}$ estimator is unbiased for the true $\ensuremath{\mathsf{GTE}}$.  On the other hand, in the $\ensuremath{\mathsf{LR}}$ design, every arriving customer considers {\em both} treatment and control listings when choosing whether to book, creating a linkage across listings through customer choice.  In other words, in the $\ensuremath{\mathsf{LR}}$ design there is interference between treatment and control, and in general the $\ensuremath{\mathsf{LR}}$ estimator will be biased for the true $\ensuremath{\mathsf{GTE}}$.%

Now return to a dynamic model where the inventory of listings is limited, and listings remain unavailable for some time after being booked.  In this case, observe that on top of the preceding discussion, there is a dynamic linkage between customers: the set of listings available for consideration by a customer is dependent on the listings considered and booked by previously arriving customers.  This dynamic effect introduces a new form of bias into estimation and is distinctly unique to our work.  In particular, because of this dynamic bias, in general the naive $\ensuremath{\mathsf{CR}}$ estimator will be biased as well.

Our paper develops a dynamic model of two-sided markets with inventory dynamics, and uses this framework to compare and contrast both the designs and estimators above.  We also introduce and study a novel class of more general designs based on {\em two-sided randomization} (of which the two examples above are special cases).  In more detail, our contributions and the organization of the paper are as follows. %

\noindent {\bf Benchmark model and formal mean field limit}.  Our first main contribution is to develop a general, flexible theoretical model to capture the dynamics described above.  In Section \ref{sec:model}, we present a model that yields a continuous-time Markov chain in which the state at any given time is the number of currently available listings of each type.  In Section \ref{sec:meanfield}, we propose a formal mean field analog of this continuous-time Markov chain, by considering a limit where the number of listings in the system and the arrival rate of customers both grow to infinity.  Scaling by the number of listings yields a continuum mass of listings in the limit.  In the mean field model, the state at a given time is the mass of available listings and this mass evolves via a system of ODEs.  Using a Lyapunov argument, we show that this system is globally asymptotically stable and give a succinct characterization of the resulting asymptotic steady state of the system as the solution to an optimization problem.  We formally establish that the mean field limit arises as the fluid limit of the corresponding finite market model, as market size grows; in other words, the mean field model is a good approximation to large markets.  The mean field model allows us to tractably analyze different estimators, as well as to study their biases in the large market regime.



\noindent {\bf Designs and estimators: Two-sided, customer-side, and listing-side randomization.}
In Section \ref{sec:experiments}, we introduce a more general form of experimental design, called {\em two-sided randomization} ($\ensuremath{\mathsf{TSR}}$); an analogous idea was independently proposed recently by \cite{bajari2019double} (see also Section \ref{sec:related}). %
In a $\ensuremath{\mathsf{TSR}}$ design,  both customers {\em and} listings are randomized to treatment and control.  However, the intervention is only applied when a treatment customer considers a treatment listing; otherwise, if the customer is in control or the  listing is in control, the intervention is not seen by the customer.  (In the example above, a customer would see the badge on a listing only if the customer were treated {\em and} the listing were treated.)  Notably, the $\ensuremath{\mathsf{CR}}$ and $\ensuremath{\mathsf{LR}}$ designs are special cases of $\ensuremath{\mathsf{TSR}}$. %
We also define natural naive estimators for each design.

\noindent {\bf Analysis of bias: The role of market balance.}  In Section \ref{sec:bias} we study the bias of the different designs and estimators proposed.  Our main theoretical results characterize how the bias depends on the relative volumes of supply and demand in the market.  In particular, in the highly {\em demand-constrained} regime (where customers arrive slowly and/or listings replenish quickly): %
{\em the naive $\ensuremath{\mathsf{CR}}$ estimator becomes unbiased, while the naive $\ensuremath{\mathsf{LR}}$ estimator is biased.}  On the other hand, in the highly {\em supply-constrained} regime (where customers arrive rapidly and/or listings replenish slowly)
we find that in fact {\em the naive $\ensuremath{\mathsf{LR}}$ estimator becomes unbiased, while the naive $\ensuremath{\mathsf{CR}}$ estimator is biased.} These findings suggest that platforms can potentially reduce bias by choosing the type of experiment based on knowledge of market balance.  %


Given the findings that $\ensuremath{\mathsf{CR}}$ and $\ensuremath{\mathsf{LR}}$ experiments offer benefits in different extremes, it is natural to ask whether good performance can be achieved in moderately balanced markets by ``interpolating'' between the naive $\ensuremath{\mathsf{CR}}$ and $\ensuremath{\mathsf{LR}}$ estimators. We define a naive $\ensuremath{\mathsf{TSR}}$ estimator that achieves this interpolation and has low bias in both market extremes, but still has large bias for moderate market balance. We then define more sophisticated $\ensuremath{\mathsf{TSR}}$ estimators that explicitly aim to correct for interference in regimes of moderate market balance. These latter estimators exhibit substantially improved performance in simulations. Appendix \ref{app:simulations} shows that these estimators perform well across a wide range of market parameters.






\noindent{\bf Insights from simulations.}
In Section \ref{sec:variance} we turn to simulations in the finite system to study the variance of the estimators. The simulations corroborate the
theoretical findings that $\ensuremath{\mathsf{TSR}}$ offers benefits with respect to bias, albeit at the cost of moderate increases in variance. Among the $\ensuremath{\mathsf{TSR}}$ estimators that we study, we find that those with smaller reductions in bias have smaller increases in variance, while those with larger reductions in bias have larger increases in variance, thus revealing a {\em tradeoff} between bias and variance for the $\ensuremath{\mathsf{TSR}}$ estimators.


In Section \ref{sec:cluster}, we compare the $\ensuremath{\mathsf{TSR}}$ approach with cluster-randomized experiments, an existing approach that platforms utilize to reduce bias.
The simulations suggest that while both approaches can reduce bias when the market is tightly clustered, $\ensuremath{\mathsf{TSR}}$ estimators can reduce bias in highly interconnected markets where cluster randomized experiments cannot. 


\vspace{0.1in}

Taken together, our work sheds light on what experimental designs and associated estimators should be used by two-sided platforms depending on market conditions, to alleviate the biases from interference that arise in such contexts.
We view our work as a starting point towards a comprehensive framework for experimental design in two-sided platforms; we discuss some directions for future work in Section \ref{sec:conclusion}.




















\section{Introduction.}\label{intro}



\input{intro}
\input{related}
\input{model}
\input{meanfield}
\input{experiments}
\input{bias}
\input{variance}
\input{cluster}
\input{conclusion}


\ACKNOWLEDGMENT{We would like to thank Navin Sivanandam from Airbnb for numerous fruitful conversations. We would also like to thank Guillaume Basse, Dominic Coey, Nikhil Garg, David Holtz, Michael Luca, Johan Ugander, and Nicolas Stier for their feedback, as well as seminar participants at ACM EC'20, Facebook, Marketplace Algorithms and Design, CODE 2020, INFORMS 2020, Lyft, Vinted, Marketplace Innovations Workshop 2021, MSOM 2021, and Stanford. Finally, we thank the ACM EC'20 review team for their thoughtful comments and engagement with our work. \gw{Complete}

This work was supported by the National Science Foundation under grants 1931696 and 1839229 and the Dantzig-Lieberman Operations Research Fellowship.}




\bibliographystyle{informs2014}

\section{A mean field model of platform dynamics}
\label{sec:meanfield}


The continuous-time Markov process described in the preceding section is challenging to analyze directly because the customers' choices and consideration sets induce complex dynamics.  Instead, to make progress we consider a formal {\em mean field} limit motivated by the regime where $N \to \infty$, in which the evolution of the system becomes deterministic.  We first present a formal mean field analogue of the Markov process introduced in the previous section and provide intuition for its derivation.  We then formally prove that the sequence of Markov processes converges to this mean field model as $N \rightarrow \infty$. The mean field model provides tractable expressions in the large market regime for  the different estimators we consider, allowing us to study and compare their bias.


The mean field model we study consists of a continuum unit mass of listings.  The total mass of listings of type $\theta$ in the system is $\rho(\theta) > 0$ (recall that $\sum_\theta \rho(\theta) = 1$).  We represent the state at time $t$ by $\v{s}_t = (s(\theta), \theta \in \Theta)$; $s_t(\theta)$ represents the mass of listings of type $\theta$ available at time $t$.  The state space for this model is:
\begin{equation}
    \label{eq:S}
 \mcal{S} = \{ \v{s} : 0 \leq s(\theta) \leq \rho(\theta) \}.
\end{equation}

We first present the intuition behind our mean field model.  Consider a state $\v{s} \in \mcal{S}$ with $s(\theta) > 0$ for all $\theta$.  We view this state as analogous to a state $\v{\sigma} \approx N \v{s}$ in the $N$'th system.  We consider the system dynamics defined by \eqref{eq:R1}-\eqref{eq:R2}.  Note that the rate at which occupied listings of type $\theta$ become available is $(m^{(N)}(\theta) - \sigma(\theta)) \tau(\theta)$, from \eqref{eq:R1}.  If we divide by $N$, then this rate becomes $(\rho(\theta) - s(\theta))\tau(\theta)$ as $N \to \infty$.  On the other hand, note that for large $N$, if $D_\gamma(\theta|\v{\sigma})$ is Binomial$(\sigma(\theta), \alpha_\gamma(\theta))$, then $D_\gamma(\theta|\v{\sigma})/N$ concentrates on $\alpha_\gamma(\theta) s(\theta)$.  Thus the choice probability $r_\gamma(\theta | \v{\sigma})$ is approximately:
\begin{equation}
\label{eq:meanfieldchoice}
p_\gamma( \theta | \v{s}) \triangleq \frac{\alpha_\gamma(\theta) v_\gamma(\theta) s(\theta) }{\epsilon_\gamma + \sum_{\theta'} \alpha_\gamma(\theta') v_\gamma(\theta') s(\theta')}.
\end{equation}
(Here we use the fact that $\epsilon_\gamma^{(N)}/N \to \epsilon_\gamma$ as $N \to \infty$.) This is the mean field multinomial logit choice model for our system.  In the finite model, the rate at which listings of type $\theta$ become occupied is $\sum_\gamma \lambda_\gamma^{(N)}  r_\gamma(\theta | \v{\sigma})$, from \eqref{eq:R2}.  If we divide by $N$, this rate becomes $\lambda \sum_\gamma \phi_\gamma  p_\gamma(\theta | \v{s})$ as $N \to \infty$.  

Inspired by the preceding observations, we define the following system of differential equations for the evolution of $\v{s}_t$:
\begin{equation}
\label{eq:ODE}
\frac{d}{dt} s_t(\theta) = (\rho(\theta) - s_t(\theta))\tau(\theta) - \lambda \sum_\gamma \phi_\gamma  p_\gamma(\theta | \v{s}_t),\ \ \theta \in \Theta.
\end{equation}

This is our formal mean field model. In the remainder of this section, we first show that this system has a unique solution for any initial condition. Then we characterize the behavior of the system. By constructing an appropriate Lyapunov function, we show that the mean field model has a unique limit point to which all trajectories converge (regardless of initial condition). This limit point is the unique steady state of the mean field limit. Finally, we prove that the sequence of Markov processes indeed converges to this mean field model (in an appropriate sense).  Hence, the mean field model provides a close approximation to the evolution of large finite markets.


\subsection{Existence and uniqueness of mean field trajectory}

First, we show the straightforward result that the system of ODEs defined in \eqref{eq:ODE} possesses a unique solution.  This follows by an elementary application of the Picard-Lindel\"{o}f theorem from the theory of differential equations.  The proof is in Appendix \ref{app:proofs}.

\begin{proposition}
\label{prop:ODE_trajectory}
Fix an initial state $\hat{\v{s}} \in \mcal{S}$.  The system \eqref{eq:ODE} has a unique solution $\{\v{s}_t:t\geq 0\}$ satisfying $0 \leq s_t(\theta) \leq \rho(\theta)$ and for all $t$ and $\theta$, and $\v{s}_0 = \hat{\v{s}}$. 
\end{proposition}




\subsection{Existence and uniqueness of mean field steady state}

Now we characterize the behavior of the mean field limit. We show that the system of ODEs in \eqref{eq:ODE} has a unique limit point, to which all trajectories converge regardless of the initial condition.  We refer to this as the {\em steady state} of the mean field system.  We prove the result via the use of a convex optimization problem; the objective function of this problem is a Lyapunov function for the mean field dynamics that guarantees global asymptotic stability of the steady state.

Formally, we have the following result.  The proof is in Appendix \ref{app:proofs}.

\begin{theorem}
\label{thm:ODE_ss}
There exists a unique steady state $\v{s}^* \in \mcal{S}$ for \eqref{eq:ODE}, i.e., a unique vector $\v{s}^* \in \mcal{S}$ solving the following system of equations:
\begin{equation}
\label{eq:FOC}
 (\rho(\theta) - s^*(\theta))\tau(\theta) = \lambda \sum_\gamma \phi_\gamma  p_\gamma(\theta | \v{s}^*),\ \ \theta \in \Theta.
\end{equation}
This limit point has the property that $0 < s^*(\theta) < \rho(\theta)$ for all $\theta$, i.e., it is in the interior of $\mcal{S}$.  Further, this limit point is globally asymptotically stable, i.e., all trajectories of $\eqref{eq:ODE}$ converge to $\v{s}^*$ as $t \to \infty$, for any initial condition $\v{s}_0 \in \mcal{S}$.  

The limit point $\v{s}^*$ is the unique solution to the following optimization problem:
\begin{align}
\text{minimize}  \ \ \ & W(\v{s}) \triangleq \sum_\gamma \left( \lambda_\gamma \log\left(\epsilon_\gamma + \sum_\theta \alpha_\gamma(\theta) v_\gamma(\theta) s(\theta) \right) \right) \nonumber \\
& \hspace{5em}- \tau(\theta) \sum_\theta \rho(\theta) \log s(\theta) + \tau(\theta) \sum_\theta s(\theta)  \label{eq:ss_objective} \\ 
\text{subject to} \ \ \ &0 \leq s(\theta) \leq \rho(\theta), \ \ \theta \in \Theta. \label{eq:ss_constraint}
\end{align}
\end{theorem}

The function $W$ appearing in the proposition statement is {\em not} convex; our proof proceeds by first noting that it suffices to restrict attention to $\v{s}$ such that $s(\theta) > 0$ for all $\theta$, then making the transformation $y(\theta) = \log (s(\theta))$.  The objective function redefined in terms of these transformed variables {\em is} strictly convex, and this allows us to establish the desired result.





\subsection{Convergence to the mean field limit}
Finally, we formally describe the sense in which our system converges to the system in \eqref{eq:ODE}. We first move from analyzing the number of listings available in the $N$'th system to analyzing the proportion of listings available. To this end, define the normalized process $\v{Y}_t^{(N)}$ where
\begin{equation*}
    Y_t^{(N)}(\theta) = \sigma_t^{(N)}(\theta)/N , \ \ \theta \in \Theta.
\end{equation*}
Note that under this definition, $\v{Y}_t^{(N)}$ is also a continuous time Markov process with dynamics induced by the dynamics of $\v{\sigma}_t^{(N)}$; in particular, the chain $\v{Y}_t^{(N)}$ has the same transition rates as $\v{\sigma}_t^{(N)}$, but increments are of size $1/N$. The following theorem establishes the convergence of $Y_t^{(N)}$ to the solution of the ODE described in \eqref{eq:ODE} as $N \rightarrow \infty$.


\begin{theorem}
\label{thm:mf_convergence}
Assume that $\epsilon_\gamma^{(N)} / N \rightarrow \epsilon_\gamma$ for all $\gamma$ and $\lambda_\gamma^{(N)} / N \rightarrow \lambda_\gamma$ for all $\gamma$ as $N\rightarrow \infty$.  Fix $\hat{\v{s}} \in \mcal{S}$ and assume that $\v{Y}_0^{(N)}$ is deterministic, with $Y_0^{(N)}(\theta)  \rightarrow \hat{s}(\theta)$ for all $\theta$. %
Let $\v{s}_t$ denote the unique solution to the system defined in \eqref{eq:ODE}, with initial condition $\v{s}_0 = \hat{\v{s}}$.

Then for all $\delta>0$ and for all times $u>0$,
\begin{equation}
    \P\left[\sup_{0 \leq t \leq u} \| \v{Y}_t^{(N)} - \v{s}_t \| > \delta \right] = O\left(\frac{1}{N}\right).
\end{equation}
\end{theorem}


The proof for this result relies on an application of Kurtz's Theorem for the convergence of pure jump Markov processes; full details are in Appendix \ref{app:proofs}.
We note that this result holds for any sequence of initial conditions $\v{Y}_0^{(N)}$, as long as the proportion of available listings at time $t=0$ converges to a constant vector $\hat{\v{s}}$ as $N \rightarrow \infty$; further, the vector $\hat{\v{s}}$ can be any (feasible) initial state in the mean field model.  %

We now utilize the mean field model to study experimental designs and interference. 








































\section{A Markov chain model of platform dynamics}
\label{sec:model}

In this section, we first introduce the basic dynamic platform model that we study in this paper with a finite number $N$ of listings.  In the next section, we describe a formal mean field limit of the model inspired by the regime where $N \to \infty$. This mean field limit model then serves as the framework within which we study the bias  of different experimental designs and associated estimators in the remainder of the paper.  

We consider a two-sided platform where we refer to the supply side as {\em listings} and the demand side as {\em customers}.     %
Customers arrive over time and at the time of arrival, the customer forms a consideration set from the set of available listings in the market and then decides whether to book one of them. If the customer books, then the selected listing 
is occupied for a random length of time during which it is unavailable for other customers. At the end of this booking, the listing again becomes available for use for other customers. %

The formal details of our model are as follows.

\textbf{Time.}  The system evolves in continuous time $t \geq 0$.

\textbf{Listings.} The system consists of a fixed number $N$ of listings.  We refer to ``the $N$'th system" as the instantiation of our model with $N$ listings present.  %
We use a superscript ``$N$" to denote quantities in the $N$'th system where appropriate.  

We allow for heterogeneity in the listings. Each listing $\ell$ has a {\em type} $\theta_\ell \in \Theta$, where $\Theta$ is a finite set (the {\em listing type space}).  Note that in general, the type may encode both observable and unobservable covariates; in particular, our analysis does not presume that the platform is completely informed about the type of each listing. For example, in a lodging site $\theta_\ell$ may encode observed characteristics of a house such as the number of bedrooms, but also characteristics that are unobserved by the platform because they may be difficult or impossible to measure.
Let $m^{(N)}(\theta)$ denote the total number of listings of type $\theta$ in the $N$'th system.  For each $\theta \in \Theta$, we assume that $\lim_{N \to \infty} m^{(N)}(\theta)/N = \rho(\theta) > 0$.  Note that $\sum_\theta \rho(\theta) = 1$.





\textbf{State description.}  At each time $t$, each listing $\ell$ can be either {\em available} or {\em occupied} (i.e., occupied by a customer who previously booked it).  The system state at time $t$ in the $N$'th system is described by $\v{\sigma}_t^{(N)} = (\sigma_t^{(N)}(\theta))$, where $\sigma_t^{(N)}(\theta)$ denotes the number of listings of type $\theta$ available in the system at time $t$. Let $S_t^{(N)} = \sum_\theta \sigma_t^{(N)}(\theta)$ be the total number of listings available at time $t$.  In our subsequent development, we develop a model that makes $\v{\sigma}_t^{(N)}$ a continuous-time Markov process.  



\textbf{Customers.} Customers arrive to the platform sequentially and decide whether to book, and if so, which listing to book.  Each customer $j$ has a {\em type} $\gamma_j \in \Gamma$, where $\Gamma$ is a finite set (the {\em customer type space}) that represents customer heterogeneity. As with listings, the type may encode both observable and unobservable covariates, and again, our analysis does not presume that the platform is completely informed about the type of each customer. %
Customers of type $\gamma$ arrive according to a Poisson process of rate $\lambda_\gamma^{(N)}$; these processes are independent across types. Let $\lambda^{(N)} = \sum_\gamma \lambda_\gamma^{(N)}$ be the total arrival rate of customers.   Let $T_j$ denote the arrival time of the $j$'th customer.  

We assume that $\lim_{N \to \infty} \lambda^{(N)}/ N = \lambda > 0$, i.e., the arrival rate of customers grows proportionally with the number of listings when we take the large market limit. Further, we assume that for each $\gamma \in \Gamma$, we have $\lim_{N \to \infty} \lambda_\gamma^{(N)}/\lambda^{(N)} = \phi_\gamma > 0$.  Note that $\sum_\gamma \phi_\gamma = 1$.


\textbf{Consideration sets}. In practice, when customers arrive to a platform, they typically form a {\em consideration set} of possible listings to book; the initial formation of the consideration set may depend on various aspects of the search and recommendation algorithms employed by the platform.   To simplify the model, we capture this process by assuming that on arrival, each listing of type $\theta$ that is available at time $t$ is included in the arriving customer's consideration set independently with probability $\alpha_\gamma(\theta) > 0$ for a customer of type $\gamma$.  For example, $\alpha_\gamma(\theta)$ can capture the possibility that the platform's search ranking is more likely to highlight available listings of type $\theta$ that are more attractive for a customer of type $\gamma$, making these listings more likely to be part of the customer's consideration set; this effect is made clear via our choice model presented below.
After the consideration set is formed, a choice model is then applied to the consideration set to determine whether a booking (if any) is made.  %



Formally, the customer choice process unfolds as follows.  Suppose that customer $j$ arrives at time $T_j$.  For each listing $\ell$, let $C_{j\ell} = 0$ if the listing is unavailable at $T_j$.  Otherwise, if listing $\ell$ is available, then let $C_{j\ell} = 1$ with probability $\alpha_{\gamma_j}(\theta_\ell)$, and let $C_{j\ell} = 0$ with probability $1-\alpha_{\gamma_j}(\theta_\ell)$, independently of all other randomness.  Then the consideration set of customer $j$ is $\{ \ell : C_{j\ell} = 1\}$.

Our theoretical results in this paper are developed with this model of consideration set formation.  Other models of consideration set formation are also reasonable, however.  As one example, customers might sample a consideration set of a fixed size, regardless of total number of listings available. We explore such a consideration set model through simulations in Appendix \ref{app:simulations} and show that similar insights hold.

\textbf{Customer choice}.  Customers choose at most one listing to book and can choose not to book at all.  We assume that customers have a {\em utility} for each listing that depends on both customer and listing types: a type $\gamma$ customer has utility $v_\gamma(\theta) > 0$ for a type $\theta$ listing. %
Let $q_{j\ell}$ denote the probability that  arriving customer $j$ of type $\gamma_j$ books listing $\ell$ of type $\theta_\ell$.

In this paper we assume that customers make choices according to the well-known {\em multinomial logit choice model}.  In particular, given the realization of $\v{C}_j$, we have:
\begin{equation}
\label{eq:mnl}
q_{j\ell} = \frac{ C_{j\ell} v_{\gamma_j}(\theta_\ell)}{\epsilon_{\gamma_j}^{(N)} + \sum_{\ell' = 1}^N C_{j\ell'} v_{\gamma_j}(\theta_{\ell'})}.
\end{equation}
Here $\epsilon_\gamma^{(N)} > 0$ is the value of the {\em outside option} for type $\gamma$ customers in the $N$'th system.  The probability that customer $j$ does not book any listing at all grows with $\epsilon_\gamma^{(N)}$.  We let the outside option scale with $N$; this is motivated by the observation that in practical settings, the probability a customer does not book should remain bounded away from zero even for very large systems.  In particular, we assume that $\lim_{N \to \infty} \epsilon_\gamma^{(N)}/N = \epsilon_\gamma > 0$.  

We note that this specification of choice model, although it relies on the multinomial logit model, can be quite flexible because we allow for arbitrary heterogeneity of listings and customers. 

For later reference, we define:
\begin{equation}
\label{eq:qjtheta}
    q_j(\theta) = \ensuremath{\mathbb{E}}\left[ \sum_{\ell : \theta_\ell = \theta} q_{j\ell}\right],
\end{equation}
where the expectation is over the randomness in $\v{C}_j$.  With this definition, $q_j(\theta)$ is the probability that customer $j$ books an available listing of type $\theta$, where the probability is computed prior to realization of the consideration set.








\textbf{Dynamics: A continuous-time Markov chain.}  
The system evolves as follows.  Initially all listings are available.\footnote{As the system we study is irreducible and we analyze its steady state behavior, it would not matter if we chose a different initial condition.}  Every time a customer arrives, the choice process described above unfolds.  An occupied listing remains occupied, independent of all other randomness, for an exponential time that is allowed to depend on the type of the listing.\footnote{An even more general model might allow the occupancy time to depend on {\em both} listing type {\em and} the type of the customer who made the booking; such a generalization remains an interesting open direction.} More formally, let $\tau>0$ and for each type $\theta$ define $\nu(\theta)$ such that, once booked, a listing of this type will remain occupied for an exponential time with parameter $\tau \nu(\theta)$.  We overload notation and define $\tau(\theta) = \tau \nu(\theta)$.  Once this time expires, the listing returns to being available.

When fixing $\v{\nu} = (\nu(\theta), \theta \in \Theta)$ and all system parameters except for $\tau$, increasing $\tau$ will make the system less supply constrained and decreasing $\tau$ will make the system more supply constrained, while preserving the relative occupancy times of each listing type.










Our preceding specification turns $\v{\sigma}_t^{(N)}$ into a continuous-time Markov process on a finite state space $\mcal{S}^{(N)} = \{ \v{\sigma} : 0 \leq \sigma(\theta) \leq m^{(N)}(\theta),\  \forall \theta \}$.  We now describe the transition rates of this Markov process.  For a state $\v{\sigma} \in \mcal{S}^{(N)}$, $\sigma(\theta)$ represents the number of available listings of type $\theta$.

There are only two types of transitions possible: either (i) a listing that is currently occupied becomes available, or (ii) a customer arrives, and books a listing that is currently available.  (If a customer arrives but does not book anything, the state of the system is unchanged.)  Let $\v{e}_\theta$ denote the unit basis vector in the direction $\theta$, i.e., $e_\theta(\theta) = 1$, and $e_\theta(\theta') = 0$ for $\theta' \neq \theta$.  The rate of the first type of transition is:
 \begin{equation}
 \label{eq:R1}
  R(\v{\sigma}, \v{\sigma} + \v{e}_\theta) = (m^{(N)}(\theta) - \sigma(\theta)) \tau(\theta),
 \end{equation}
since there are $m^{(N)}(\theta) - \sigma(\theta)$ booked listings of type $\theta$, and each remains occupied for an exponential time with mean $1/\tau(\theta)$, independently of all other randomness.  

The second type of transition requires some more steps to formulate.  In principle, our choice model suggests that the identity of both the arriving guest and individual listings affect system dynamics; however, our state description only tracks the aggregate number of listings of each type available at each time $t$.  The key here is that our entire specification depends on customers {\em only} through their type, and depends on listings {\em only} through their type.  

Formally, suppose a customer $j$ of type $\gamma_j = \gamma$ arrives to find the system in state $\v{\sigma}$.  For each $\theta$ let $D_\gamma(\theta | \v{\sigma})$ be a Binomial$(\sigma(\theta), \alpha_\gamma(\theta))$ random variable, independently across $\theta$.   Recall that for each available listing $\ell$, each $C_{j\ell}$ is a Bernoulli$(\alpha_\gamma(\theta_l))$ random variable.  Recalling $q_j(\theta)$ as defined in \eqref{eq:qjtheta}, it is straightforward to check that:
\begin{equation}
\label{eq:anonymizedchoice}
 q_j(\theta) = r_\gamma(\theta | \v{\sigma}) \triangleq \ensuremath{\mathbb{E}} \left[\frac{ D_\gamma(\theta|\v{\sigma}) v_{\gamma}(\theta)}{\epsilon_{\gamma}^{(N)} + \sum_{\theta'} D_\gamma(\theta'|\v{\sigma}) v_{\gamma}(\theta')}\right].
\end{equation}
In other words, the probability an arriving customer of type $\gamma$ books a listing of type $\theta$ when the state is $\v{\sigma}$ is given by $r_\gamma(\theta | \v{\sigma})$; and this probability depends on the past history {\em only} through the state $\v{\sigma}$ (ensuring the Markov property holds).  

With this definition at hand, for states $\v{\sigma}$ with $\sigma(\theta) > 0$, the rate of the second type of transition is:
\begin{equation}
\label{eq:R2}
 R(\v{\sigma}, \v{\sigma} - \v{e}_\theta) = \sum_\gamma \lambda_\gamma^{(N)}  r_\gamma(\theta | \v{\sigma}).
\end{equation}

Note that the resulting Markov chain is irreducible, since customers have positive probability of sampling into, and booking from, their consideration set, and every listing in the consideration set has positive probability of being booked.  %



\textbf{Steady state.} 
Since the Markov process defined above is irreducible on a finite state space, there is a unique steady state distribution $\pi^{(N)}$ on $\mcal{S}^{(N)}$ for the process.  %
























\section{Response to Reviewer \thereviewer}}

\newcommand{\sectionrev}[1]{\bigskip \hrule
\section{{#1}}}

\newenvironment{pointEditor}
{\begin{sf} \refstepcounter{pointEditor} \bigskip \noindent { \textbf{Point~\thepointE} } ---\ }
	{\par \end{sf}}

\newenvironment{pointAE}
{\begin{sf} \refstepcounter{pointAE} \bigskip \noindent { \textbf{Point~\thepointAE} } ---\ }
	{\par  \end{sf}}

\newenvironment{point}
{\begin{sf} \refstepcounter{point} \bigskip \noindent {\textbf{Point~\thepoint} } ---\ }
	{\par  \end{sf}}

\newcommand{\shortpoint}[1]{\refstepcounter{point}  \bigskip \noindent 
	{\begin{sf}\textbf{Point~\thepoint}  ---~#1\par  \end{sf}}}

\newenvironment{reply}
{\medskip \noindent \textbf{Reply}:\  }
{\medskip}

\newcommand{\shortreply}[1][]{\medskip \noindent \textbf{Reply}:\ #1 
	\medskip}
	
	


\title{Experimental Design in Two-Sided Platforms: An Analysis of Bias\\
\textit{Authors' Response to Management Science Reviews}}

\date{\vspace{-3em}}
\author{}
\usepackage[round, sort]{natbib}

\begin{document}

\maketitle

\section{Summary of changes}
\label{sec:summary}

The authors would like to thank the reviewers for their feedback.
Motivated by their suggestions, we have made several changes to better engage with both the existing literature on experiments with interference as well as the use of these experiments in practice. 

With respect to the existing literature on methods to reduce bias, we have added a section comparing the bias reduction of TSR designs with that of the existing cluster-randomized designs. We show that cluster-randomized designs lead to lower bias when the market is tightly clustered, but TSR designs lead to lower bias in more interconnected markets. Regarding the use of these experiments in practice, we added discussions on robustness to our modeling assumptions, the application of these proposed experiment types to different kinds of interventions one may want to test in practice, and the open questions on statistical inference in these settings.
We also provide guarantees on the sign of the bias in some specific market settings, which platforms can use to heuristically bound the true global treatment effect. 

Additionally, we have updated the simulation figures to better allow the reader to compare the performance of the experiments across different market scenarios and we have added 95th percentile bootstrapped intervals to better illustrate the simulation error. 

We provide detailed replies to editor, AE, and referees below. =


\section{Response to editor}
Dear David:  Thanks very much for a thorough and constructive review process. We also very much appreciate the encouragement with a minor revision recommendation. Our revision has followed the suggestions from the review team closely. As a result, we think the paper has further improved. 

As you pointed out the main request was to compare with other approaches by using the simulations. We have added extensive simulations to address the issues raised by the referees; further details are provided below. Indeed, we provide detailed replies to the AE and referees.

We look forward to hearing back from you. 




\sectionrev{Response to AE}

We would like to thank you for the very constructive and thoughtful reviews. We really appreciate your encouragement and that you think our work tackles a relevant problem and makes important contributions. Our revision has followed your suggestions and those from the review team closely. As a result, we think the paper has further improved. Below, we describe how we have addressed your individual comments. For your convenience, we include your comments.



\begin{pointAE}
1. Comparison with cluster-randomized designs.

Reviewer 1 raises the question of how the proposed experimental designs compare with others in the literature. This seems like an important question to me. What would happen if other designs were used in these settings? Even if not developed with this particular data generating process in mind, they might perform reasonable well — or not. Analytical results may be beyond the scope of this paper, but it seems valuable to include these in the simulations.

Thinking about this also highlighted how some other approaches end up being inapplicable precisely because of (for many applications) unrealistic versions of the model: if customers and listings are all homogeneous, then there is no structure to exploit in a clustered design. But of course this is not realistic for many applications, whether ride sharing (where the market is segmented by geography and car type), lodging, or general freelancing. So this comparison would need to allow for meaningfully heterogeneous customers and listings (currently these simulations are in the appendix) and have some idea of how the clustered designs use this (typically this is through past interactions).

\end{pointAE}
\begin{reply}
In this version of the paper we added a section on comparisons between TSR and cluster-randomized experiments. To do so, we introduce a utility model with two customer types and two listing types that is parametrized to capture how tightly the market is clustered. Via numerical simulations we show that when the market is tightly clustered, cluster-randomized experiments offer substantial bias improvements. However, as there is more overlap between clusters, TSR estimators outperform cluster-randomized estimators. While our model for market clusters  is stylized, our results suggest that TSR experiments can be a useful alternative to cluster-randomized experiments. Please see Section \ref{sec:cluster} and Appendix \ref{app:cluster} for more details.
\end{reply}

\begin{pointAE}
2. Sign of the bias
Reviewer 2 raises a questions about signing the bias in the model in the simulations. This seems quite interesting because this would allow bounding the true estimand under weaker assumptions. Other work on interference has taken an interest in such partial identification (e.g. Choi 2017, Eckles et al. 2016, Pouget-Abadie et al. 2018). My sense is that the kind of interference due to congestion that is possible here can straightforwardly produce bias across the null, which can occur in some models of contagion (Morozova et al. 2018) but is explicitly ruled out by others (prior references). More generally, perhaps the authors can help readers gain intuition about the bias, and they may want to make sure the simulations illustrate what they regard as a relevant range of biases. (I liked the stylized extremes in the simple examples in the analytical results as one way they currently do this.)
\end{pointAE}
\begin{reply}
We now show that for a broad class of positive treatment effects, the sign of the bias for LR and CR in the respective extremes of market balance is always positive (and negative for negative treatment effects). This is intuitive as cannibalization will induce an over estimation of bookings in the treatment group relative to global treatment, and an underestimation of bookings in the control group relative to global control. We confirm the robustness of these results via simulations for intermediate regimes of market balance. %
\end{reply}


\newpage

\begin{pointAE}
3. Statistical inference

One thing that is absent from the paper is any discussion of statistical inference (i.e., confidence intervals, hypothesis tests). Typical uses of randomized experiments require assessing statistical uncertainty. The simulations involve reporting standard errors, but these are the true (i.e. infeasible) standard errors computed through replications of the simulation. The authors should at least note that statistical inference is unaddressed here, but perhaps they can do a little more. It should be straightforward to test a sharp null hypothesis of no effects whatsoever (Fisher's null) using Fisherian randomization inference. I wonder then whether the Type I error rate (size) of these tests is in the simulations. I think it is reasonable for a full treatment of statistical inference to be beyond the scope of this paper, but I'd also suggest that the paper's impact (especially on empirical research and industry practice) will be much greater if there are least some pointers towards inference.
\end{pointAE}
\begin{reply}

We agree that statistical inference is an important direction to study, particularly with regard to practical impact of this work.  At the same time, there are significant challenges in this direction.  Notably, the idea of deriving valid confidence intervals in these settings is complicated by the fact that the point estimates of the GTE are themselves biased. Even if we could perfectly estimate the standard errors, the bias in the point estimate could lead to invalid confidence intervals as well. Thus statistical inference involves a combination of debiasing the treatment effect estimate as well as the standard error estimate.

While we don't address this topic in the current manuscript, motivated by your suggestion we have now added a discussion in Section \ref{sec:conclusion} highlighting open questions in this area.  We discuss how platforms typically estimate standard errors ``naively,'' assuming independence across observations, which is clearly violated in these settings with interference and results in possible bias in the standard error estimates as well.


\end{reply}

\begin{pointAE}
4. Presentation of simulation results.

Some of the reviewer comments may help the authors improve the presentation of the results, perhaps making some of the results more readily understood. For example, both Reviewers 1 \& 2 seem to have compared related results in Figures 4-11 — with some difficulty. Perhaps it is possible to make this clearer. One could imagine combining the panels that have the same outcome into a single (e.g.) line graph. I will leave this to the authors, but I think many readers will want to make these comparisons.
\end{pointAE}
\begin{reply}
We have added figures that compare the performance of the estimators across different market conditions and levels of heterogeneity, in addition to the existing plots that compare across levels of supply and demand imbalance. Please see Appendix \ref{app:simulations}.
\end{reply}

\begin{pointAE}
I also wanted to compare MSE of estimators, but could not readily do so.
\end{pointAE}
\begin{reply}
Thanks for the suggestion. We have added plots for RMSE of the estimators. 
\end{reply}

\begin{pointAE}
I would suggest that the authors provide some guidance about the Monte Carlo error in all of these results. Maybe error bars are not needed everywhere, but as noted by Reviewer 1 it can be hard to tell, e.g., which estimators/designs are indistinguishable from unbiasedness.
\end{pointAE}
\begin{reply}
Thank you for your suggestion. To provide guidance on the simulation error, we have added 95 percentile bootstrap confidence intervals to all simulation plots.
\end{reply}

\begin{pointAE}
In summary, this paper provides a nice analysis of an increasingly important problem. Comparison with existing designs seems important for quantifying the advantages of the proposed design, and the authors might consider some further improvements for clarity and guidance for applying this (where statistical inference will be needed).
\end{pointAE}
\begin{reply}
We appreciate the positive feedback, and we have incorporated both the suggestions for comparison against existing (cluster randomized) designs and clarity in the figures and exposition. 
\end{reply}


	


\reviewersection

We would like to thank you for the careful reading of our paper and your encouragement. We were pleased to learn that you think we are studying an important problem and that we made good progress replying to the EC reviews. We have followed your suggestions in this revision. 
As a result, we think the paper is more streamlined and reads better. Below, we describe how we have addressed your individual comments. For your convenience, we include your comments.

\begin{point}
My largest concern with the paper is that the current draft is a bit hard to follow. I’ve spent a lot of time with this manuscript, and still feel unsure about some important points. I think there are a few things the authors can do to streamline the paper and make it easier to understand.

• I think the paper is currently a bit overstuffed; some of the examples in the paper feel redundant, and some of the ideas in the paper seem underdeveloped and distracting from the main point in their current form. On the example front, the examples provided in Sections 6.1 and 6.2 feel very similar to the example in Section 7.1.3 (the authors even note in 7.1.3, “we consider a setting analogous to Section 6.”). My guess is that the examples in Section 6 are supposed to be toy examples that build intuition, whereas the examples in Section 7.1.3 are meant to be an application of the mathematical tools built up earlier in Section 7, but it ends up feeling like the paper is repeating itself. Maybe it is possible to move one of these sets of examples to the appendix?
\end{point}
\begin{reply}
Thank you for the comment. We have removed the toy examples and instead use the application of our model to the homogeneous setting in Section \ref{ssec:Qij_homogeneous} (previous Section 7.1.3) to build intuition. 
\end{reply}

\begin{point}
 I’m also not sure the discussion of transient effects belongs in the main body of the paper. I think this is an important topic, given that firms are often in the position of making long-term product decisions based on short experiments that run for weeks, if not days. However, the discussion of transient estimators here seems under-developed. What should my takeaway be after reading Section 7.3 and looking at Figure 2? What are the implications, both in terms of different estimators and in terms of thinking about short-term vs. long-term experiments? Given that the transient vs. steady state contrast is not key to the arguments in the paper, and the authors do not devote a great amount of time to comparing, say, the rate at which bias of different estimators approaches 0 while in the transient system, I’m wondering if this analysis is better suited to an appendix.
\end{point}
\begin{reply}
That's a good point. We have moved the transient discussion to the appendix. 
\end{reply}

\newpage
\begin{point}
 When the TSR design is first presented in Section 5, it feels to me like it comes a bit out of left field. Yes, the LR and CR designs are mathematically special cases of the TSR design, but that doesn’t provide me any intuition for why the TSR design should actually be less biased than CR or LR, so as a reader I end up getting sidetracked trying to figure this out. When presenting the TSRI designs later in the paper, I think the authors do a better job providing intuition for what the different terms in the estimator are doing (e.g., “linear combination of LR and CR, with terms that account for cannibalization”). I’d love to see more of that type of intuition throughout the paper when it comes to the TSR designs and the corresponding estimators.
\end{point}
\begin{reply}
We have provided more motivation for TSR when we first introduce it in Section 5 and when we introduce it as an interpolation between CR and LR. We have also provided better intuition of the cannibalization terms for TSRI in Section \ref{ssec:TSR_est} and Figure \ref{fig:tsr_comp_terms}.%

\end{reply}

\begin{point}
I recognize this is appendix content, but I was quite interested in the performance of the TSR estimators in simulations as market conditions were varied (e.g., different levels of heterogeneity, etc). In order to try and reason about this, I found myself trying to compare numbers in Figures 6, 7, and 8, numbers in Figures 9, 10, and 11, etc. I wonder if these bar graphs are the right way to communicate these results? For instance, I could imagine Figures 6, 7 and 8 being combined into one figure, where rather than relative demand on the x-axis, the level of heterogeneity is on the x-axis, and the relative demand is fixed for a given row of results. Not sure how this would look, so just a suggestion. But I think my broader point is that having these results split across multiple figures (with different y-axes in some cases) makes the natural comparisons hard to do.
\end{point}
\begin{reply}
Thanks for this comment. We have added figures that compare the performance of the estimators across different market conditions at a fixed level of supply and demand (Appendix \ref{ssec:robustness_market_scenarios}), in addition to the existing plots that compare across levels of supply and demand imbalance (Appendix \ref{ssec:robustness_vary_lambda}). We now have four figures that depict the effect of changing average utility, customer heterogeneity, listing heterogeneity, and treatment effect heterogeneity in a scenario with balanced supply and demand. 
\end{reply}

\begin{point}
In addition to the above points about writing clarity, it would be great to see the paper connect a bit more directly to the existing literature on experimentation in two-sided marketplaces, which the authors reference in the literature review (Holtz 2018, Holtz et al. 2020, Sneider et al. 2019). Is it possible to calculate the bias of cluster randomized estimators, or switchback estimators, using the analytical framework developed in this paper? If not, is it possible to at least code up these estimators in the simulations used in the back half of the paper, so that cluster randomized experiments and switchback experiments can be included in the bias and variance comparisons? It would be helpful to know how TSR compares not just to LR and CR designs (which we know have bias issues, but are pretty good on the variance front), but also previously proposed solutions (which we know are at least a bit better on bias, but often much worse in terms of variance). In case it’s helpful, I think one way to represent cluster randomized experiments in your framework would be to model consumer choice using the type of nested choice model that you sometimes see in empirical IO, and then assign treatment and control at the level the groups in that nested model.
\end{point}
\begin{reply}
This is a great comment. In this version of the paper we added a section on comparisons between TSR and cluster-randomized experiments. To do so, we introduce a utility model with two customer types and two listing types that is parametrized to capture how tightly the market is clustered. Via numerical simulations we show that when the market is tightly clustered, cluster-randomized experiments offer substantial bias improvements. However, as there is more overlap between clusters, TSR estimators outperform cluster-randomized estimators. While our model for market clusters  is stylized, our results suggest that TSR experiments can be a useful alternative to cluster-randomized experiments. Please see Section \ref{sec:cluster} and Appendix \ref{app:cluster} for more details.
\end{reply}


\begin{point}
Finally, I think it’s important for the authors to discuss some of the limitations of the framework presented and experiment designs proposed in this paper. For one, the stochastic market model proposed by the authors in many ways simplifies the dynamics that we would observe in an actual marketplace. For instance, as the authors note in footnote 4, their model allows the length of a booking to depend on the type of listing, but not the type of customer. Of course, simplifications like this are required in order to do this type of work. That being said, I think it is important to spend at least a small amount of time in the discussion section of the paper discussing these assumptions, and some potential ways in which they might threaten the validity of the paper’s results when applied to real world scenarios (or, if the authors feel confident that none of these assumptions threaten the validity of the conclusions reached, a convincing argument making this point would be great also). 
\end{point}
\begin{reply}
Thank you for this note. We have added a discussion on robustness of our results to modeling assumptions in Section \ref{sec:conclusion}. We believe that our core insights of market balance mediating competition
effects, and thus affecting bias in an experiment, broadly extends to other settings. We conjecture that some of our results are more robust to modeling choices than others. For example, the result that the CR estimator is unbiased in the demand constrained regime may be more robust than the result that the LR estimator is unbiased in the supply constrained regime.
\end{reply}

\begin{point}
Similarly, I can think of a number of treatment interventions a two-sided platform might want to test that are not well-suited to the TSR design. For instance, if a platform wants to test a new search ranking algorithm, it is not clear to me how this change would only be applied to some listings, but not others, i.e., the TSR design as proposed does not seem possible. This isn’t meant to take away from the potential benefits of TSR – it seems like a helpful design. It would just be helpful for the authors to discuss scenarios where the TSR design is, and is not, appropriate.

• Related to my comment above about not all treatments lending themselves to TSR, one of the examples provided on Page 14 (changing the frequency with which the recommendation system promotes listings of a certain type) seems like one such broken example. Even if I only treat some of the would-be promoted listings, isn’t this going to by definition “treat” some of the other listings who are not being promoted (assuming there is some type of UI space constraint)?
\end{point}
\begin{reply}
This is a very good observation. The TSR design is more suited to interventions that act on a single customer-listing pair. The ranking algorithm you mention would act on a customer and a set of listings, which would not be well suited to the TSR design. We now provide a discussion of this in Section \ref{sec:conclusion}. On the second point you bring up about the recommendation setting example, we agree that if there is a UI space constraint or, additionally, a constraint on customer attention, then changing one listing would affect others. This is another reason to further study modifications in the consideration set formation, where perhaps customers can only sample a fixed number of listings into their consideration set. We briefly study this scenario in Appendix \ref{app:simulations} but leave further study on the choice model for future work. For the time being, we have removed the recommendation system example and replaced it with an example where the platform reduces friction for a listing in the checkout flow. 
\end{reply}

\begin{point}
Minor points
\end{point}
\begin{reply}
We have addressed all of your minor points. We list the most prominent ones below.
\end{reply}

\begin{point}
• The paper really hits the ground running, which I didn’t mind as someone familiar with the literature on experimentation in two-sided platforms, but it might be good to ease the reader into this literature a bit more in the introduction, and explain why this is an important problem. I could see someone less familiar with the existing research asking “OK, so sometimes these experiments yield biased estimates – so what?”
\end{point}
\begin{reply}
We have provided a bit more context in the introduction where we now discuss how experiments are used and previous literature on the magnitude of the bias in the marketplace experiments.
\end{reply}

\begin{point}
• When varying customer and listing heterogeneity in Appendix B, the authors change the utilities, but I’m wondering if there’s a way to look at how the bias/variance changes as the number of customer types or listing types changes? This might not be easy to do, since this might also depend on the probabilities of inclusion in the choice set, etc., but when I think of heterogeneity, I think of the number of types, whereas what the authors do now seems to be something more akin to heterogeneous treatment effects.
\end{point}
\begin{reply}
We have added simulations where we increase the heterogeneity of the listings and/or customers before the intervention. Appendix \ref{app:simulations} now contains simulations where we fix the number of listings types, but vary pre-treatment utilities that customers have for these types to consider varying levels of heterogeneity between the listings. Further, we fix the relative lift that treatment has on the utilities to be uniform across all types, to remove any heterogeneous treatment effects. We find that this heterogeneity affects the bias of the CR estimator, but remarkably does not affect the bias of the TSR estimators. We also provide a similar analysis where we vary the heterogeneity between customers and show that this heterogeneity affects the bias of the LR estimator, but again not the TSR estimators.

\end{reply}

\begin{point}
• The consideration set analysis in the appendix that was added after EC is useful. I’d be curious to see how the bias/variance of different estimators change as the size of the consideration sets change. Perhaps it is worth redoing the included analysis for a couple of additional values of K (e.g., 10, 50, 100, 500).
\end{point}
\begin{reply}
We believe that modifications in customer behavior (including the size of the consideration set, how customers sample listings into the set, and the choice model used) are all important directions to study and require further analysis than we can provide with our current model. We leave much of this direction to future work in the area. 
\end{reply}

\begin{point}
• It may be worth doing a comparison of the simulation RMSEs, in addition to showing bias and variance separately. Granted, RMSE is just $bias^2 + variance$, but it may be tough for the reader to do that calculation in their head, especially when the y-axes in the subfigures are on different scales.
\end{point}
\begin{reply}
We have now added RMSE of the estimators to all simulation figures. We remark in Section \ref{sec:variance} that the performance of these estimators with respect to RMSE depends largely on the size of the platform (or the time horizon of the experiment). The bias of the experiment is relatively stable with changes in these two factors, but of course variance decreases in the size and the length of the time horizon. Thus whether bias or variance contributes more to the RMSE depends on the scale of the market and experiment. 
\end{reply}

\begin{point}
Again, I really enjoyed reading this paper, and am glad to see people doing work in this area. Thank you, and good luck!
\end{point}
\begin{reply}
Thanks again for your constructive comments and encouragement.
\end{reply}


\reviewersection

We would like to thank you for the careful reading of our paper your comments. We have followed your suggestions in this revision. 
As a result, we think the paper has further improved. Below, we describe how we have addressed your individual comments. For your convenience, we include your comments.



\begin{point}
On page 28, the authors mentioned that "the estimators are all upward-biased because of the parameter values chosen." I was wondering if there could be any guarantee on the sign of the bias due to interference, for the logit choice model studied in this paper. In practice, I could imagine that the exact size of the treatment effect is not the most critical for decision making: a company might be comfortable launching a new feature, if the estimation is positive, and they know that the estimator is guaranteed to underestimate a positive treatment effect.

From the results presented here, it seems like interference generally leads to overestimating a positive treatment effect. I tried to think about this but couldn't easily find an example going in the other direction. In Figure 2 in the original submission to EC the TSR estimators were underestimating in transient phase, but the new Figure 2 in the current submission is showing a different dynamic. 
\end{point}
\begin{reply}
We now show that for a broad class of positive treatment effects, the sign of the bias for LR and CR in the respective extremes of market balance is always positive (and negative for negative treatment effects). This is intuitive as cannibalization will induce an over estimation of bookings in the treatment group relative to global treatment, and an underestimation of bookings in the control group relative to global control. We confirm the robustness of these results via simulations for intermediate regimes of market balance. %
\end{reply}

\begin{point}
Thanks for including the simulation results on page 47 where the size of the consideration set is fixed at K=50. In comparison to earlier figures, the biases seem to be substantially larger (>0.8 vs around 0.02 for oversupplied CR, for example). I wonder if there is any intuitive explanations for this. 
\end{point}
\begin{reply}
This is a good observation, and unfortunately one for which our current analysis using the mean field model isn't sufficiently equipped to answer. However, we believe that modifications in customer behavior (including the size of the consideration set, how customers sample listings into the set, and the choice model used) are all important directions to study and require further analysis than we can provide with our current model. We leave much of this direction to future work in the area. 
\end{reply}

\begin{point}
With $\lambda / \tau = 10$, would the platform run out of listings, and would the actual consideration set be smaller than 50 in this case?
\end{point}
\begin{reply}
When the platform has fewer than 50 listings, the customer will sample all listings available into the consideration set. We have modified the text in Appendix \ref{app:simulations} to make this clear. 
\end{reply}

\begin{point}
I found myself wondering why are the correction terms in (40) multiplied by $\beta$ and $(1-\beta)$ respectively
\end{point}
\begin{reply}
The motivation is to weight the correction term less when the corresponding type of competition is weaker. For example, as we approach the highly demand-constrained regime, the term corresponding to customer competition gets diminishing weight since we know that type of interference vanishes in that regime. A symmetric argument applies for the supply-constrained regime and other competition term. 

It is possible that the correction terms themselves will go to 0 as the competition weakens, in which case the $\beta$ and $(1-\beta)$ weights would be redundant, but we leave the optimization of the $\ensuremath{\mathsf{TSR}}$ estimator for future work. 
\end{reply}

\begin{point}
I probably missed this, but for Figure 2, is the initial state the stationary state equilibrium when all customers and listings are in control?
\end{point}
\begin{reply}
In the earlier version, the initial steady state for the transient numerics was a market with all listings available. However, motivated by your question, we have realized that it is more practical for the initial steady state (in the experiment) to be the steady state equilibrium when all customers and listings are in control. We have now modified the figure to start at the steady state for global control.
\end{reply}


\begin{point}
It could be helpful if the y axis are normlized for the first two subfigures in Figures 4-11. 
\end{point}
\begin{reply}
Thank you for the suggestion. We have now normalized the axes in the mean field bias and simulation bias figures. See Appendix \ref{app:simulations}. We find that the mean field bias and simulation bias are quite close in each of the scenarios we consider. 
\end{reply}

\begin{point}
Also, for a number of settings, we see that the mean field bias is substantially smaller than the simulation bias for TSRI2 but not for TSRI-1, and I am curious why do we see this.
\end{point}
\begin{reply}
For settings with few customers $(\lambda = 0.1)$, we see that in some cases the simulation bias in TSRI-2 differs more from the mean field bias than the the other estimators. In some cases (small average utility) the simulation bias is lower, while in other cases (large heterogeneity between customers) the simulation bias is larger. We believe that these discrepancies are due to the larger standard errors associate with the TSRI-2 estimator. 

In this revised manuscript, we ran larger simulations than in the original submission ($N=5000$ instead of $N=1000$) and the discrepancies between the mean field and simulation bias are smaller. In addition, we have added 95 percentile bootstrapped intervals to each of the simulation plots, and we can see that the mean field bias lies within the corresponding interval. 
\end{reply}


\newpage

\begin{point}
I did find the left panel in Figure 1 of the original EC (bias ~ $\lambda / \tau$ for different estimators) submission very helpful, and I'm also curious what this figure would look like if normalized by GTE
\end{point}
\begin{reply}
We have included figures where we give mean field and simulation results for the estimators $\lambda \in \{0.1, 1, 10\}$. 

Thanks again for all the constructive comments.
\end{reply}

\reviewersection
\begin{point}
I am satisfied with the changes the authors have made to the manuscript.
\end{point}

We really appreciate your encouragement and support of our work. 



\end{document}

\section{Related work}
\label{sec:related}


{\bf SUTVA.} The types of interference described in these experiments are violations of the Stable Unit Treatment Value Assumption (SUTVA) in causal inference  \citep{ImbensRubin15}.  SUTVA
requires that the (potential outcome) observation on one unit should be unaffected by the particular assignment of treatments to the other units.t
A large number of recent works have investigated experiment design in the presence of interference, particularly in the context of markets and social networks. 

\noindent {\bf Interference in marketplaces.} Biases from interference can be large: \cite{Blake14} empirically show in an auction experiment that the presence of interference among bidders caused the estimate of the treatment effect to be wrong by a factor of two. \cite{Fradkin_2019} finds through simulations that a marketplace experiment changing search and recommendation algorithms can overestimate the true effect by 50 percent. %
More recent work by \cite{holtz2020reducing} randomizes clusters of similar listings to treatment or control, and finds the bias due to interference can be almost 1/3 of the treatment effect.  Interestingly, \cite{holtz2020reducing} also finds weak empirical evidence that the extent of interference depends on market balance; our paper provides strong theoretical grounding for such a claim.

Inspired by the goal of reducing such bias, other work has developed approaches to bias characterization and reduction both theoretically (e.g., \cite{Basse16} in the context of auctions with budgets), as well as via simulation (e.g., \cite{Holtz18} who explores the performance of $\ensuremath{\mathsf{LR}}$ designs).  Our work complements this line, by developing a mathematical framework for the study of estimation bias in dynamic platforms.  Key to our analysis is the use of a mean field model to model both transient and steady-state behavior of experiments.  A related approach is taken in \cite{Wager19}, where a mean field analysis is used to study equilibrium effects of an experimental intervention where treatment is incrementally applied in a marketplace (e.g., through small pricing changes).



\noindent {\bf Interference in social networks.}  A bulk of the literature in experimental design with interference considers an interference that arises through some underlying social network: e.g., \cite{Manski13} studies the identification of treatment responses under interference; \cite{Ugander13} introduces a graph cluster based randomization scheme and analyzes the bias and variance of the design; and many other papers, including  \cite{Athey18, Basse19, Saveski17} focus on estimating the spillover effects created by interference. In particular, \cite{pouget2019variance} and \cite{zigler2018bipartite} consider interference on a bipartite network, which is closer to a two-sided marketplace setting. In general, this line of work considers a fixed interference pattern (social network) over time. Our work is distinct because the interference caused by supply and demand competition is endogenous to the experiment and dynamically evolving over time. 

  

\noindent {\bf Other experimental designs.}  In practice, platforms currently mitigate the effects of interference through either clustering techniques that change the unit of observation to reduce spillovers among them (e.g., \cite{chamandy16}), similar to some of the works mentioned above (e.g., \cite{Holtz18, Ugander13}); or by {\em switchback testing} \citep{sneider19}, in which the treatment is turned on and off over time.  Both cause a substantial increase in estimation variance due to a reduction in effective sample size and thus the naive $\ensuremath{\mathsf{CR}}$ and $\ensuremath{\mathsf{LR}}$ designs remain popular workhorses in the platform experimentation toolkit. In addition to these broad classes of experiments, other work has also introduced modified experiment designs for specific types of interventions, such as \cite{hathuc2020counterfactual} for ranking experiments. 

\noindent {\bf Two-sided randomization.} Finally, a closely related paper is \cite{bajari2019double}.  Independently of our own work, there the authors propose a more general multiple randomized design of which $\ensuremath{\mathsf{TSR}}$ is a special case.  They focus on a static model and provide an elegant and complete statistical analysis under a local interference assumption.  By contrast, we focus on a dynamic platform model with market-wide interference patterns and focus on a mean field analysis of bias.


\subsection{Transient behavior}

For practical implementation, it is important to consider the relative bias in the candidate estimators in the transient system, since experiments are typically run for relatively short time horizons. 
Theoretically, we can provide some insight when $\tau \to \infty$: in this case, the dominant term in the right hand side of \eqref{eq:ODE} is $(\rho(\theta) - s_t(\theta))\tau$.  Using this fact, it can be shown that as $\tau \to \infty$, for each $t > 0$, there holds $\v{s}_t^*(a_C, a_L) \to \v{\rho}(a_L)$ (where we define $\v{\rho}(a_L)$ as in Section \ref{ssec:ss_theory}).  In other words, the state {\em remains} at $\v{\rho}(a_L)$ at all times.   As a result in this limit the transient estimators $\widehat{\ensuremath{\mathsf{GTE}}}^{\ensuremath{\mathsf{CR}}}(T | a_C)$ and $\widehat{\ensuremath{\mathsf{GTE}}}^{\ensuremath{\mathsf{LR}}}(T | a_L)$ are equivalent to $\widehat{\ensuremath{\mathsf{GTE}}}^{\ensuremath{\mathsf{CR}}}(\infty | a_C)$ and $\widehat{\ensuremath{\mathsf{GTE}}}^{\ensuremath{\mathsf{LR}}}(\infty | a_L)$, respectively.  In particular, asymptotically as $\tau \to \infty$, the transient estimator $\widehat{\ensuremath{\mathsf{GTE}}}^{\ensuremath{\mathsf{CR}}}(T | a_C, a_L)$ will be an {\em unbiased} estimate of $\ensuremath{\mathsf{GTE}}$ at all times $T > 0$.  (The same is true if $\lambda \to 0$, provided the initial state is $\v{s}_0 = \v{\rho}(a_L)$.)

More generally, Figure \ref{fig:transient_dynamics} numerically investigates how the time horizon of the experiment affects the performance of the estimators in the mean field model; there we plot an example where $\lambda/\tau$ takes a moderate value.  The relative performance of estimators depends on the time horizon of interest as well as market balance conditions. 

\begin{figure}[H]
    \centering
    \begin{tabular}{c c c }
        \includegraphics[height=0.26\textwidth]{figs_transient/dynamics_lam_01_mod.png}
        &
        \includegraphics[height=0.26\textwidth]{figs_transient/dynamics_lam_1_mod.png}
        & 
        \includegraphics[height=0.26\textwidth]{figs_transient/dynamics_lam_10_mod.png}
    \end{tabular}
    \caption{Transient dynamics of estimators with $\lambda/\tau = 0.1, 1$, and $10$. Parameters are as defined in Figure \ref{fig:numerics}.}
    \label{fig:transient_dynamics}
\end{figure}






\section{Bias-variance tradeoff of estimators}
\label{sec:variance}



Our mean field model is deterministic, so it does not allow us to study the {\em variance} of the different estimators. In practice, however, markets consist of finitely many listings and experiments are run for finite time horizon $T$, and so the variance of any estimator will be nonzero.\footnote{We note that even if the system only consists of a finite number of listings $N$, as $T \to \infty$ the standard error of the various estimators proposed in this paper will converge to zero.  However, for finite $T$, this is not the case; since A/B tests are always run to a finite horizon $T$, this nonzero variance will impact the accuracy of any estimates obtained.}
In particular, the variance of estimators becomes an important consideration alongside bias, particularly in choosing between multiple estimators with similar bias.  The variance of the $\ensuremath{\mathsf{TSR}}$ estimators is especially important, given the earlier discussion that many heuristics that platforms use to minimize bias do so at the cost of increased variance, leading to under-powered experiments  (see Section \ref{sec:related}).

With this background as motivation, in this section we provide a preliminary yet suggestive simulation study of variance.  %
The simulations highlight two important considerations that a platform must take into account when designing and analyzing an experiment. First, similar to the results from Section \ref{sec:bias} on bias, we find that
the estimator with the lowest variance depends on market balance. Second, we see a bias-variance tradeoff between the $\ensuremath{\mathsf{CR}}$, $\ensuremath{\mathsf{LR}}$ and $\ensuremath{\mathsf{TSR}}$ estimators, with the $\ensuremath{\mathsf{TSR}}$ estimators offering bias improvements at the cost of an increase in variance. 
We emphasize the point that whether a platform should care more about bias or variance depends on the size of the platform (number of listings $N$) and the time horizon on which the experiment is run. The bias of the experiment is relatively unaffected by changes in these two factors, but of course variance decreases in the size of the market and the length of the time horizon. Thus these two factors dictate whether bias or variance contribute more to the overall $\mathrm{RMSE}$. 


Full details of the simulation environment and parameters are in Appendix \ref{app:simulations}, which we briefly summarize here.  We simulate marketplace experiments with varying market parameters for a finite system with a number of listings $N=5000$ and fixed time horizon $T$. For each run of the simulation, we fix an experiment design (e.g., $\ensuremath{\mathsf{CR}}$, $\ensuremath{\mathsf{LR}}$, $\ensuremath{\mathsf{TSR}}$) and simulate customer arrivals and booking decisions until time $T$.  System evolution is simulated according to the continuous time Markov chain specified in \eqref{eq:R1}-\eqref{eq:R2}. We calculate the estimator corresponding to the experiment design, defined in \eqref{eq:naive_CR}-\eqref{eq:naive_TSR_est} and \eqref{eq:TSRI-A} (for $k=1,2$) for the time interval $[T_0, T]$, where $T_0$ is chosen to eliminate the transient burn-in period.  We then simulate multiple runs and compare the bias and standard error of the estimators across runs. 
Note that we report the true standard errors, calculated across simulation runs. For discussion on the estimation of standard errors, see Section \ref{sec:conclusion}.


Figure \ref{fig:simulations_hom} shows simulations for a homogeneous system with only one customer and one listing type, with the same parameters as the mean field numerics presented in Figure \ref{fig:numerics_homogeneous}. Note that the bias of the estimators in these large market simulations echo the qualitative insights about bias obtained from the mean field model. Similar findings are obtained in more general scenarios, cf.~Appendix \ref{app:simulations}, where we investigate the effect of heterogeneity in the marketplace.



\begin{figure}
    \centering
        \includegraphics[height=0.3\textwidth ]{figs_sims/homogeneous_listings_customers_bias_simulations_error_bar_mod.png}
        \includegraphics[height=0.3\textwidth]{figs_sims/homogeneous_listings_customers_se_simulations_error_bar_mod.png}
        \\
        \hspace{2cm}\includegraphics[height=0.3\textwidth]{figs_sims/homogeneous_listings_customers_rmse_simulations_error_bar_mod.png}

    \caption{(Homogeneous listings and customers.) Top Left: Bias of each estimator. 
    Top right: Standard error of estimates. 
    Bottom: $\mathrm{RMSE}$ of the estimates. Statistics are normalized by $\ensuremath{\mathsf{GTE}}$. All statistics are calculated across 500 runs, with bootstrapped 95 percentile confidence intervals provided for each statistic.
    We consider a setting with homogeneous listings and customers, with the same utilities as defined in Figure \ref{fig:numerics_homogeneous}. In the $\ensuremath{\mathsf{CR}}$ design, $a_C = 0.5$. In the $\ensuremath{\mathsf{LR}}$ design, $a_L = 0.5$.  Simulation parameters are defined in Appendix \ref{app:simulations}.
    }
    \label{fig:simulations_hom}
\end{figure}


These simulations point to a bias variance-tradeoff between $\ensuremath{\mathsf{TSR}}$ estimators and the naive $\ensuremath{\mathsf{CR}}$ and $\ensuremath{\mathsf{LR}}$ estimators, as well as between the three $\ensuremath{\mathsf{TSR}}$ estimators themselves. The $\ensuremath{\mathsf{TSR}}$ estimators, as discussed earlier, offer benefits over the naive $\ensuremath{\mathsf{CR}}$ and $\ensuremath{\mathsf{LR}}$ estimators with respect to bias, but they do so at the cost of an increase in variance. Moreover, among the three $\ensuremath{\mathsf{TSR}}$ estimators that we explore, those with lower bias also have higher variance.  The naive $\ensuremath{\mathsf{TSRN}}$ estimator has similar variance with the lowest of $\ensuremath{\mathsf{CR}}$ and $\ensuremath{\mathsf{LR}}$, but the bias of this estimator is also similar to the lowest bias of $\ensuremath{\mathsf{CR}}$ and $\ensuremath{\mathsf{LR}}$. On the other hand, $\ensuremath{\mathsf{TSRI\text{-}2}}$ shows a substantial improvement in bias over both $\ensuremath{\mathsf{CR}}$ and $\ensuremath{\mathsf{LR}}$ for several market conditions, but this estimator also has the largest variance among all five estimators, especially in the regime of intermediate market balance (cf. Appendix \ref{app:simulations}).

Further, the minimum variance estimator depends on market conditions. For example, in a demand-constrained market with $\lambda/\tau = 0.1$, $\widehat{\ensuremath{\mathsf{GTE}}}^{\ensuremath{\mathsf{CR}}}$ has the lowest standard error, whereas in a supply-constrained market with $\lambda/\tau=10$, $\widehat{\ensuremath{\mathsf{GTE}}}^{\ensuremath{\mathsf{CR}}}$ has the highest standard error. 


We conclude this section by highlighting the potential for the class of $\ensuremath{\mathsf{TSR}}$ experiments, which open up a large class of both designs and estimators. Among the three estimators we explore, we see that there is a $\ensuremath{\mathsf{TSR}}$ estimator with low bias and another $\ensuremath{\mathsf{TSR}}$ estimator with low variance. It is possible that, with further optimization of these designs and estimators, one can devise a new estimator that optimizes this bias-variance tradeoff. %








2021-09-28T02:27:22.000
2002
2002.05670
en
https://arxiv.org/abs/2002.05670
We develop an analytical framework to study experimental design in two-sided marketplaces. Many of these experiments exhibit interference, where an intervention applied to one market participant influences the behavior of another participant. This interference leads to biased estimates of the treatment effect of the intervention. We develop a stochastic market model and associated mean field limit to capture dynamics in such experiments, and use our model to investigate how the performance of different designs and estimators is affected by marketplace interference effects. Platforms typically use two common experimental designs: demand-side ("customer") randomization (CR) and supply-side ("listing") randomization (LR), along with their associated estimators. We show that good experimental design depends on market balance: in highly demand-constrained markets, CR is unbiased, while LR is biased; conversely, in highly supply-constrained markets, LR is unbiased, while CR is biased. We also introduce and study a novel experimental design based on two-sided randomization (TSR) where both customers and listings are randomized to treatment and control. We show that appropriate choices of TSR designs can be unbiased in both extremes of market balance, while yielding relatively low bias in intermediate regimes of market balance.
Methodology (stat.ME); Econometrics (econ.EM)
Experimental Design in Two-Sided Platforms: An Analysis of Bias
Qwen/Qwen-72B
1. YES
2. YES