--- abstract: 'In many important applications of precision medicine, the outcome of interest is time to an event (e.g., death, relapse of disease) and the primary goal is to identify the optimal individualized decision rule (IDR) to prolong survival time. Existing work in this area have been mostly focused on estimating the optimal IDR to maximize the restricted mean survival time in the population. We propose a new robust framework for estimating an optimal static or dynamic IDR with time-to-event outcomes based on an easy-to-interpret quantile criterion. The new method does not need to specify an outcome regression model and is robust for heavy-tailed distribution. The estimation problem corresponds to a nonregular M-estimation problem with both finite and infinite-dimensional nuisance parameters. Employing advanced empirical process techniques, we establish the statistical theory of the estimated parameter indexing the optimal IDR. Furthermore, we prove a novel result that the proposed approach can consistently estimate the optimal value function under mild conditions even when the optimal IDR is non-unique, which happens in the challenging setting of exceptional laws. We also propose a smoothed resampling procedure for inference. The proposed methods are implemented in the R-package `QTOCen`. We demonstrate the performance of the proposed new methods via extensive Monte Carlo studies and a real data application.' author: - 'Yu Zhou, Lan Wang, Rui Song and Tuoyi Zhao' bibliography: - 'censorref.bib' title: '**Transformation-Invariant Learning of Optimal Individualized Decision Rules with Time-to-Event Outcomes**' --- \[1\] <span> Yu Zhou is a machine learning xbowers@example.org. Email: xbowers@example.org. Lan Wang is Professor, Department of Management Science, University of Miami. Email: xbowers@example.org. Rui Song is Professor, Department of Statistics, North Carolina State University. Email: xbowers@example.org. Tuoyi Zhao is a Ph.D. student, Department of Management Science, University of Miami. Wang, Zhou and Zhao's research was supported by NSF FRGMS-1952373. Song's research was supported by NSF DMS-2113637. We thank Co-Editor Professor McKeague, the AE and the anonymous referees for their helpful comments. </span> <span>**KEY WORDS:**</span> Exceptional laws, individualized decision rule, inference, precision medicine, robust method, time-to-event data. Introduction ============ The problem of estimating the optimal individualized decision rule (IDR) has recently received substantial attention in precision medicine and other domains. A treatment can be a drug, a therapy, or any other actionable choice (or a sequence of such choices) such as a policy or program. The goal of optimal IDR estimation is to determine a decision rule that assigns a subject to one of the treatment options based on individual information available each decision point such that some functional of the potential outcome distribution is optimized. For completely observed data, several successful approaches exist for estimating the optimal IDR, including Q-learning [@watkins1992; @Murphy05; @chak2010; @Song2015], A-learning [@robins2000; @Murphy2003; @murphy2005experimental; @moodie2010], model-free or policy search methods [@Robins2008; @Orellan10; @zhang2012robust; @zhao2012estimating; @zhao2015], the interpretation-enhanced tree or list-based methods [@laber2015tree; @cui2017tree; @zhu2017greedy; @Zhang2017Lists] among others. See also the books of @CM2013 and @KM2016 for a general introduction and additional references. The focus of this paper is on estimating the optimal static (one-stage) or dynamic (multi-stage) IDR for time-to-event or survival data, where the outcome is possibly censored. Several new challenges arise when analyzing such data compared with the complete data case. Censoring occurs when an individual drops out from the study or the study ends before the subject experiences the event of interest. The distribution of survival time (e.g., time to death, onset of disease) is often highly skewed. The situation gets even more complicated for estimating the optimal dynamic IDR, where the data are collected longitudinally. Consider the setting where the treatment decisions for a patient are made at $k$ pre-specified decision points. Each decision is allowed to depend on the patient's characteristics (e.g., gender, age) and treatment history (e.g., disease progression status and how the individual responds to previous treatments) up to that decision point. The patient may be censored at any stage of treatment. Direct application of existing complete-data techniques could result in severe bias, as demonstrated in the Monte Carlo studies in Section 5. Several authors have recently investigated estimating the optimal IDR with survival data, see @goldberg2012q, @xu2016bayesian, @jiang2017estimation, @jiang2017doubly, @bai2017optimal, @hager2018optimal, @xu2016bayesian, @diaz2018targeted, @simoneau2019, among others. For time-to-event outcomes, new criterion is needed to evaluate the effectiveness of an IDR. Of crucial importance is that such a criterion can be reliably estimated under censoring. The existing work have been mostly focused on maximizing the <span>*restricted mean survival time*</span> in the population. We consider time-to-event outcomes and propose a new robust framework for estimating the optimal IDR using an alternative criterion based on the marginal quantile of the potential outcome distribution. The new optimality criterion is easy to interpret. Median survival time has already been popularly used clinically to evaluate the success of cancer treatment. It can be reliably estimated even under relatively heavy censoring (e.g., the censoring rate is more than 50% in the real data example of this paper). The resulted optimal IDR is invariant under monotone-transformation of the outcome. We develop robust estimation methods for both static and dynamic optimal IDRs. The robust approach circumvents the difficulty of specifying a reliable outcome regression model, especially for the dynamic setting which demands a sequence of generative regression models. We consider estimating the optimal IDR in a class of candidate decision rules indexed by a finite-dimensional parameter. The estimation problem corresponds to a challenging non-regular M-estimation problem with both finite and infinite-dimensional nuisance parameters, due to the unknown censoring distribution. For the optimal static IDR, we rigorously establish the cube-root convergence rate for the estimated parameter by employing modern empirical processes techniques. We prove that its asymptotic distribution corresponds to the maximizer of a centered Gaussian process with a parabolic drift. It is worth emphasizing that the nonstandard asymptotics is due to the intrinsic nature of the decision problem, which relates to a sharp edge effect in the decision function. Due to the nature of nonstandard asymptotics, the problem is substantially harder than regular M-estimation problem with infinite-dimensional nuisance parameters. Moreover, we establish a useful novel result that shows the optimal value can be consistently estimated under weak conditions for the challenging setting of exceptional laws. Under exceptional laws, there exists a subgroup of patients for whom the treatment is neither beneficial nor harmful. In this case, the optimal IDR is non-unique, see for example, the discussions in @robins2014discussion and @Luedtke2016. Theoretically, our work complements existing results and significantly enhances the knowledge about optimal IDR estimation with survival data. The existing results have been mostly focused on prediction error bounds and have not studied the properties of the estimated parameter indexing the optimal IDR. Furthermore, to the best of our knowledge, existing work on optimal IDR estimation with survival outcomes assume non-exceptional laws and thus avoid the problem of optimal value estimation under exceptional laws. The rest of the paper is organized as follows. Section 2 introduces the new framework and the robust method for estimating an optimal static IDR with survival data, with theoretical properties developed in Section 3. Section 4 presents the estimation method and the theory for the dynamic IDR setting. In Section 5, we report results from extensive Monte Carlo studies. In Section 6, we illustrate the application on the analysis of a breast cancer data set. The proposed methods are implemented in the R-package `QTOCen`. The regularity conditions are given in the Appendix. The technical derivations and additional numerical results are given in the online supplement. Robust estimation for static optimal IDR ========================================= Preliminaries ------------- We first consider the single-stage setting. Let $ A \in \{0, 1\}$ denote the binary treatment, $ \vX \in \mathbb{R}^p $ denote the vector of covariates with support $ \mathcal{X}$, and $ T\in \mathbb{R}^{+} $ denote the time to the event of interest (or a transformation thereof). We often refer to $T$ as <span>*survival time*</span>. Without loss of generality, we assume that a larger value of $T$ indicates better treatment effect. The outcome $T$ may not be observed due to censoring. Let $C \in \mathbb{R}^{+}$ denote the censoring variable and $\Delta = I (T \leq C )$. If the observation is censored (i.e., $\Delta=0$), then we only observe $C$. Let $Y=\min\{T, C\} $ be the observed outcome. The observed data consist of $\big\{\vX_i,A_i,Y_i,\Delta_i \big\}$, $i=1, \ldots, n$, which are independent copies of $ \big(\vX,A,Y,\Delta \big) $. To assess the treatment effect, we adopt the potential outcome framework [@neyman1923applications; @rubin1978bayesian] in causal inference. The $i$-th subject has two potential outcomes: $ \TstarZeroi$ and $\TstarOnei $, where $T_i^*(0)$ is the survival time had the subject received treatment 0 and $T_i^*(1)$ is defined similarly, $i=1,...,n$. In practice, a subject receives one and only one of the two possible treatments. Under the stable unit treatment value assumption (@Rubin86), we have $ T_i=A_i\TstarOnei+\big(1-A_i\big) \TstarZeroi. $ That is, the survival time of the $i$th subject is the potential survival time corresponding to the treatment that subject actually received. Furthermore, we assume the no unmeasured confounders assumption [@RR83] is satisfied, i.e., $\{T_i^*(0), T_i^*(1)\}$ are independent of $A_i$ conditional on $\vX_i$. This is a common assumption in causal inference and is automatically satisfied for a randomized trial. Mathematically, an IDR $ d(\vX) $ is a mapping from the space of covariates $ \mathcal{X} $ to the space of candidate treatments $\{0,1\}$. The potential outcome associated with $ d(\vX) $ is denoted by $\Tstardx$. We have $ \Tstardx =\TstarOne d\big( \vX\big) + \TstarZero \big[1-d\big( \vX\big)\big]. $ New optimality criterion for evaluating IDR with time-to-event outcome {#subsection: new_criterion} ---------------------------------------------------------------------- Different from the completely observed data case, the mean of the outcome is usually difficult to estimate accurately at the presence of censoring. The most popular criterion in the existing literature for time-to-event data is the restricted mean survival time $\mbox{E}\{ \min \big(\Tstardx, L\big)\}$, where $L$ is a user-supplied cut-off time. In this paper, we consider an alternative criterion for comparing IDRs for time-to-event outcomes based on the marginal quantile treatment effect. The new criterion enjoys three appealing properties: easy to interpret, robust to long-tailed survival distribution, and invariant to the monotone transformation of survival time. The marginal $\tau$-th quantile ($0<\tau<1$) of the potential outcome $\Tstardx$ is defined as $$Q_{\tau}\big\{\Tstardx\big\} = \inf \Big\{ t\in\mathbb{R}: P \big\lbrace \Tstardx \leq t \big\rbrace \geq \tau\Big\},$$ where $P$ denotes the marginal distribution of $\Tstardx$. In IDR estimation problems, $Q_{\tau}\big\{\Tstardx\big\}$ is referred to as the <span>*value function*</span>. We sometimes use the short-hand notation $ Q_{\tau}\{T^{\ast}(d)\}$. Given a class $ \mathcal{D}$ of candidate IDRs, the optimal IDR is defined as $$\label{eq:general dopt} d_{opt}(\Tstardx)=\underset{d\in\mathcal{D}}{\arg\max} \ Q_{\tau}\big\{ \Tstardx \big\}.$$ As an example, the above optimal IDR with $\tau=0.5$ will lead to the maximal median of the potential survival time if each individual in the population follows the treatment recommended. In some other applications (e.g., distributing aid in a social welfare program), we may target improving the lower quantile by considering a smaller value of $\tau$. For an arbitrary monotone transformation $h(\cdot)$, it is known that $h\big( Q_{\tau}\{h(\Tstard) \} \big) = Q_{\tau}\{h(\Tstard)\}$ [@koenker2005quantile]. Hence, the same decision will be reached no matter the analysis is based on the original survival time or its monotonic transformation. For complete data, quantile criterion was studied in @wang2017quantile but their work is not applicable to survival data, which is also theoretically substantially harder with the presence of an infinite-dimensional nuisance parameter due to censoring. @wahed2009estimation studied estimating the survival quantiles in two-stage randomization designs with fixed IDRs but had not investigated the more challenging problem of optimal IDR estimation. In practice, $\mathcal{D}$ is usually chosen to be a class of IDRs indexed by a Euclidean parameter for interpretability. Same as @zhang2012robust and others, we focus on the class of index rules $\mathcal{D}= \Big\lbrace \dbetax = \mbox{I}\big( \bbeta^{{T}}\vX>0\big):\ \ |\beta_{1}| = 1,\ \widetilde{\bbeta} \in \mathbb{B} \Big\rbrace, $ where $\bbeta = (\beta_1, ...,\beta_p )^T=(\beta_1, \widetilde{\bm{\beta}}^T )^T$, $\mathbb{B}$ is a compact subset of $ \mathbb{R}^{p-1} $ and $\mbox{I}(\cdot)$ denotes the indicator function. For identifiability, we assume there exists a continuous covariate whose coefficient has absolute value one. Without loss of generality, we assume $|\beta_1| = 1$. The population parameter $\bbeta_0=(\beta_{01}, ...,\beta_{0p} )^T$ indexing the optimal IDR is $$\bbeta_0 =\underset{\bbeta\in\mathbb{B}^o}{\arg\max} Q_{\tau}\{\Tstardx\},$$ where $\mathbb{B}^o = \{\bbeta \in \mathbb{R}^p: |\beta_{1}| = 1,\ \tilde{\bbeta} \in \mathbb{B} \}$. A robust estimation procedure {#sec:estimation procedure} ----------------------------- Based on the observations $\big\{\vX_i,A_i,Y_i,\Delta_i \big\}$, $i=1, \ldots, n$, our goal is to estimate the population parameter $\bbeta_0$ indexing the optimal IDR. It is known that a misspecified generative regression model can result in severe bias in estimating the optimal treatment [@qian2011; @zhang2012robust; @zhao2012estimating; @zhao2015doubly]. We introduce a robust estimator that accounts for censoring while at the same time circumvents the difficulty of specifying a reliable generative regression model. Given an IDR $d_{\bbeta}(\vX)$, the treatment it would recommend to subject $i$ may or may not coincide with the treatment the subject actually received. Even if $A_i = d_{\bbeta}(\vX_i)$, we may not observe $\Tstardbetai$ if the subject is censored. To obtain a consistent estimator for $ Q_{\tau}\{ \Tstardbetai \}$, we adapt the induced missing data framework in @zhang2012robust to time-to-event data. Specifically, we consider an artificial missing data structure with the missing data indicator $ \Rbetai = \big[A_{i} \dbetaxi + \big(1-A_{i}\big)\big\lbrace 1- \dbetaxi \big\rbrace \big]\Delta_{i}. $ The observed outcome $Y_i$ is equal to the potential outcome $\Tstardbetai$ only if $\Rbetai = 1$. In this framework, the "full data" that we may not completely observe consist of $\{\vX_{i}, \Tstardbetai\}_{i=1}^n$, and the observed data consist of $ \big\{\vX_{i}, \Rbetai, \Rbetai \Tstardbetai \big\} = \big\{\vX_{i}, \Rbetai, \Rbetai Y_{i} \big\}_{i=1}^n$. Let $\pi_{A}(\vX_i)=P(A_i=1|\vX_i)$ be the propensity score; and let $ \Gtrue \big(t|\vX, A \big) = P\big(C > t | \vX, A\big) $ denote the conditional survival function of $ C$ given $\{\vX, A\}$. Let $\pi_{i}\big(\bbeta\big) = {P} \bigl\{ \Rbetai = 1 \mid \vX_{i},\TstarOnei, \TstarZeroi \bigr\}$ be the probability of missingness conditional on the full data. We observe $$\begin{aligned} &\pi_{i}\big(\bbeta\big) & \\ =\ & \pi_{A}(\vX_i)\dbetaxi {P} \bigl[ \Delta_{i}=1 \mid \vX_{i},\TstarOnei, \TstarZeroi, A_{i} =1 \bigr] & \\ & +(1-\pi_{A}(\vX_i))(1-\dbetaxi) {P} \bigl[ \Delta_{i}=1 \mid \vX_{i},\TstarOnei, \TstarZeroi, A_{i} =0 \bigr] & \\ =\ & \big\{\pi_{A}(\vX_i)\dbetaxi+(1-\pi_{A}(\vX_i))(1-\dbetaxi)\big\} \Gtrue \Big( \Tstardbetai \big| \vX, A_i=\dbetaxi \Big). & \end{aligned}$$ Note that for the complete cases (corresponding to $ \Rbetai = 1$), we have $Y_i=\Tstardbetai$ and the corresponding $$\pi_{i}\big(\bbeta\big)= \ \big[ \pi_{A}(\vX_i)\dbetaxi+ \big(1-\pi_{A}(\vX_i)\big) \big(1-\dbetaxi \big) \big] \Gtrue \big( Y_i \big| \vX, A_i \big).$$ To estimate $\bbeta_0$, we propose the following two-step estimator. First, we estimate $\Qtaubetax $ by the following inverse probability weighted estimator $$\label{eq:Qhat} \QHattaubetax = \arg\min\limits _{\substack{b}} \sum_{i=1}^{n} \frac{\ R_{i}\big( \bbeta \big)}{\widehat{\pi}_i} \rho_{\tau}\big(Y_{i} - b\big),$$ where $\widehat{\pi}_i$ is an estimate of $\pi_i(\bbeta)$ and $ \rho_{\tau}\big(u\big) = u\{\tau - I\big(u < 0\big)\}$ is the quantile loss function [@koenker2005quantile]. By convention, we define $0/0=0$. Next, employing the policy-search idea, we estimate $\bbeta_0$ by $$\label{eq: main_regime_estimator_full} \widehat{\bbeta}_n = \ \underset{\bbeta \in \mathbb{B}^o}{\arg\max} \QHattaubetax.$$ The above estimator can be computed using the genetic algorithm in R package `rgenoud` [@Mebane2011]. The estimate of the optimal IDR is $d_{\widehat{\bbeta}_n}\big(\vX\big) = I\big(\widehat{\bbeta}_n^T\vX > 0\big)$.\ *Remark 1.* A key quantity in estimating $\pi_{i}\big(\bbeta\big)$ is the conditional survival function of the censoring variable $\Gtrue \big(t|\vX, A \big) = P\big(C>t | \vX, A \big)$. There are several approach for estimating $\Gtrue \big(t|\vX, A \big)$. For clarity of presentation, as in @goldberg2012q and @jiang2017estimation, we assume that $C \perp \{\TstarZero, \TstarOne, A, \vX\}$ in the theoretical development. This is often satisfied in real applications where administrative censoring occurs. In this case, $\Gtrue \big(t|\vX, A \big)$ can be estimated by $\widehat{G}_C(\cdot)$, the classical Kaplan-Meier estimator applied to $\{ (Y_i, 1-\Delta_i), i=1,2,...,n\}$. When necessary, we can relax the independent censoring assumption to the conditionally independent censoring assumption $C\perp \{\TstarZero, \TstarOne\} \big| \{\vX, A\}$ and employs the local Kaplan-Meier estimator. Without loss of generality, we assume that the first $n_1$ subjects receive treatment $A=0$, and the other $(n-n_1)$ subjects receive treatment $A = 1$. The local Kaplan-Meier estimator @gonzalez1994asymptotic for $\Gtrue \big(\cdot|\vX, A=0 \big)$ is given by $$\label{eq: local_KM_A=0} \widehat{G}_C\big(\cdot\mid \vX, A=0\big)=\prod_{j=1}^{n_{1}}\Big\{ 1-\dfrac{B_{n_1j}\big( \vX \big)}{\sum_{k=1}^{n_{1}}I\big(C_{k}\geq C_{j}\big)B_{n_1k}\big( \vX \big)}\Big\} ^{\eta_{j}(t)},$$ where $\eta_{j}(t)=I\big(C_{j}\leq t,\Delta_{j}=0\big)$, and $\big\{ B_{n_1k}\big( \vX \big),k=1,...,n_{1}\big\} $ is a sequence of non-negative weights adding up to 1. A popular choice is the Nadaraya-Watson's type weights for univariate covariate: $ B_{nk}\big(X\big)= \big[\sum_{i=1}^{n}K\big(h_{n}^{-1}(X-X_{i})\big)\big]^{-1}K\big(h_{n}^{-1}(X-X_{i})\big) $, where $K(\cdot)$ is a positive kernel function and $h_{n}$ is a sequence of bandwidths converging to zero as $n\rightarrow\infty$. We can obtain $\widehat{G}_C\big(\cdot\mid \vX, A=1\big)$ similarly. A third approach is to estimate the conditional survival function using a working model, such as the Cox proportional hazards regression model, as investigated in @zhao2015doubly.\ *Remark 2.* The Kaplan-Meier estimator $\Ghat$ in (\[eq:Qhat\]) is sometimes unstable at the tail of the distribution. Practically, a simple approach to improve the stability is by employing an artificial censoring technique in @Zhou2006, based on the intuition that any alteration of a random variable's distribution beyond the quantile of interest would have no impact on the quantile. Specifically, assume there exists a large positive constant $ M \in \mathbb{R}$ such that $\sup_{\bbeta}\Qtaub < M$ and $\sup\{t: G_{C}(t)>0 \} >M$. The first requirement means the largest achievable $ \tau $th quantile using IDRs belonging to $ \mathcal{D} $ is smaller than $M$; and the second one ensures every data point has a positive probability of not being censored. Note that these conditions are weak, especially if we are interested in lower quantiles. Let $Y^M = Y\land M$ and $\Delta^M = \Delta + \big(1-\Delta\big)I\big(Y\geq M\big)$. Then it is straightforward to show that $\widehat{\bbeta}^M_n $ obtained using the transformed data set $ \big(\vX_i, A_i, Y^M_i,\Delta^M_i\big) $, $ i=1,\ldots,n$, is the same as $\widehat{\bbeta}_n $ in (\[eq: main\_regime\_estimator\_full\]).\ *Remark 3.* As the optimization problem is nonconcave and nonsmooth, multiple local optimal may exist. Popular algorithms based on derivatives do not work for this challenging setting. At the same time, it is impractical to exhaustively enumerate all possible solutions and pick the best one. In our numerical experiments, we utilize the genetic algorithm in the R package `rgenoud`, which is useful in such a challenging setting when the objective function is nonconcave and the derivatives do not exist. The genetic algorithm (a type of evolutionary algorithm) is inspired from the biological evolution process. In a genetic algorithm, the problem is encoded in a series of bit strings that are manipulated by the algorithm. It is a stochastic, population-based algorithm that searches randomly by mutation and crossover among population members. It is based on searching for the best solutions using inheritance and strengthening of useful features of multiple objects of a specific application in the process of imitation of their evolution. We refer to @mitchell1998 and @Mebane2011 for more detailed description of the algorithm and other references. In our numerical experience, the algorithm provides high-quality solutions with desirable statistical properties. Asymptotic theory {#sec: statistical properties of single-stage estimator} ================= In this section, we present two results regarding the asymptotic properties of the estimated optimal IDR with survival data. - First, we show that the estimated parameter indexing the optimal IDR has nonstandard asymptotics, which is characterized by the cube-root convergence rate and the non-normal limiting distribution. - Second, we show that under rather weak conditions, which do not require the optimal IDR to be unique at the population or sample level, the theoretically optimal value can be estimated at a near $n^{-1/2}$-rate. Both results are novel for optimal IDR estimation with time-to-event outcomes. The first result corresponds to a nonstandard estimation problem with both finite-dimensional and infinite-dimensional estimation problem. The second result deals with the challenging setting of exceptional law where the optimal IDR is nonunique. Asymptotic distribution of the estimated parameter indexing the optimal IDR {#sec31} --------------------------------------------------------------------------- Write $\vX = (X_1, ..., X_p)^T \equiv (X_1, \tilde{X}^T)^T$. Let $\Gtrue(\cdot)$ denote the survival function of $C$. Let $ F_{T^{\ast}(0)}(t|\vX)$ and $F_{T^{\ast}(1)}(t|\vX)$ be the cumulative distribution functions of the potential survival times $T^{\ast}\big(0\big)$ and $T^{\ast}\big(1\big)$, respectively; and let $ f_0(t \vert \vX) $ and $ f_1(t \vert \vX) $ be the corresponding conditional density functions. Given any $\bbeta \in \mathbb{B}^p$, let $ f_{\Tstardbeta}(\cdot) $ denote the marginal density function of the distribution of the potential survival time $T^{\ast}(\dbetax)$. To avoid complications irrelevant to the main results of the paper, we consider data collected from a randomized study where $\pi_{A}(\vX_i)=0.5$, but the results can be extended to observational data under mild assumptions. The estimator $\widehat{Q}_{\tau}\big(\Tstardbeta\big)$ in (\[eq:Qhat\]) simplifies to $$\label{eq:QtauEstimator_Ghat2} \widehat{Q}_{\tau}\big(\bbeta; \Ghat\big) = \arg\min\limits _{\substack{b}} \sum_{i=1}^{n} \frac{\ R_{i}\big( \bbeta \big)\ \rho_{\tau}\big(Y_{i} - b\big) } {0.5 \Ghat\big(Y_i\big)},$$ where $\Ghat(\cdot)$ is the classical Kaplan-Meier estimator of $\Gtrue(\cdot)$. We then estimate $\bbeta_0$ by $\widehat{\bbeta}_n = \underset{\bbeta \in \mathbb{B}^o}{\arg\max} \widehat{Q}_{\tau}\big(\bbeta; \Ghat\big)$. Write $\widehat{\bbeta}_n = (\widehat{\beta}_{n1}, \widetilde{\bbeta}_n^T)^T$, where $\widehat{\beta}_{n1}$ satisfies the identifiability condition $|\widehat{\beta}_{n1}| = 1.$ In the proof of Theorem \[thm: asymptotic\_dis\_of\_beta\_n\] in the online supplement, it was shown that $\widehat{\bbeta}_n$ is consistent for $\bbeta_0$. Hence, we have $\widehat{\beta}_{n1} = \beta_{01}$ with probability approaching one. Theorem \[thm: asymptotic\_dis\_of\_beta\_n\] below states the nonstandard convergence rate and non-normal limiting distribution of $\widehat{\bbeta}_n$. \[thm: asymptotic\_dis\_of\_beta\_n\] Suppose conditions (C1)-(C4) are satisfied. Then as $n\rightarrow \infty$, $$n^{1/3} \big(\widetilde{\bbeta}_n - \widetilde{\bbeta}_{0}\big) \rightarrow \arg\max\limits _{\substack{t}} \big\{ \Psi(t) + \mathbb{W}(t) \big\}$$ in distribution, where $ \Psi (t) $ is a deterministic function whose form is given in (17) of the online supplement and $ \mathbb{W}(t) $ is a mean-zero Gaussian process with covariance function given in (19) in the online supplement. The proof of Theorem \[thm: asymptotic\_dis\_of\_beta\_n\] is given in the online supplement. Theoretical analysis of the asymptotic distribution of $\widehat{\bbeta}_n$ in (\[eq:QtauEstimator\_Ghat2\]) is challenging, as it is defined via a bilevel optimization problem. The proof involves reformulating $\widehat{\bbeta}_n$ as an $M$-estimator with a nonsmooth objective function that has two nuisance parameters: a finite dimensional nuisance parameter $m_0$ and an infinite dimensional nuisance parameter $\Gtrue(\cdot)$. The nonstandard asymptotics arise from the so-called *sharp-edge effect*, see @KimPollard90 for an informative example of the shorth estimator that illustrates this phenomenon. It is worth noting that the theory in @KimPollard90 can only handle a finite dimensional nuisance parameter, hence is not applicable in our setting.\ *Remark 4.* Our results are related to recent work on non-standard estimation problem in @banerjee2007, @sen2010inconsistency, @matsouaka2014evaluating, @wang2017quantile, @shi2018massive, @patra2018consistent and @banerjee2019divide. However, none of the above work involves an infinite-dimensional nuisance parameter as we face in the current setting. In fact, our estimation method involves both finite-dimensional and infinite-dimensional nuisance parameters, the role of the latter is for estimating the censoring distribution. Due to the nature of nonstandard asymptotics, the problem is different from and much harder than regular M-estimation problem with infinite-dimensional nuisance parameters. Advanced empirical process techniques from @van1996, @kosorok2008 and @delsol2015semiparametric were adapted here to help deal with the theoretical challenges.\ Estimating the optimal value with possibly non-unique optimal IDR {#subsec_3-optimalvaluetheory} ----------------------------------------------------------------- Besides the optimal IDR itself, a quantity of interest is the optimal value, defined as $$\label{define_Vopt} V_{opt} = \underset{\bbeta \in \mathbb{B}^o}{\sup}\ \Qtaubetax.$$ This quantity is the maximally achievable marginal $\tau$-th quantile of the potential distribution of all IDRs in the given class of candidate rules. It is an important measure of the performance of the optimal IDR. A natural estimator of this quantity is $\widehat{V}_n = \widehat{Q}_{\tau} \big(\widehat{\bbeta}_n; \widehat{G}_C \big)$. Theorem \[thm: limit\_V\] shows that under rather weak conditions, $\widehat{V}_n$ provides a near $n^{-1/2}$-rate estimator for $V_{opt}$. To avoid the requirement of a unique optimal IDR, the derivation of this result is based on recognizing that the following equivalent expressions: $$\label{Vopt_equivalent} V_{opt} = \sup \Big\{v : \underset{\bbeta \in \mathbb{B}^o}{\sup} P g\big(\cdot, \bbeta, v, \Gtrue\big) \geq 1-\tau \Big\},$$ $$\widehat{V}_{n} = \sup \Big\{v : \underset{\bbeta \in \mathbb{B}^o}{\sup} P_n g\big(\cdot, \bbeta, v, \Ghat \big) \geq 1-\tau \Big\},$$ where $$\label{shark} g\big(\cdot, \bbeta, v, \Gtrue\big) = \frac{R(\bbeta)}{0.5 \Gtrue(Y)} I(Y-v >0).$$ The function $g\big(\cdot, \bbeta, v, \Gtrue\big)$ is motivated by the first-order optimization condition of the estimator $\widehat{Q}_{\tau} \big(\bbeta; \widehat{G}_C \big)$. A careful inspection reveals that (\[define\_Vopt\]) and (\[Vopt\_equivalent\]) are equivalent. To see this, we observe that for a given $\bbeta$, $g\big(\cdot, \bbeta, v, G\big)$ is a monotonically decreasing function of $v$, and $Pg\big(\cdot, \bbeta, v, G\big) \geq 1-\tau$ when $v \leq \Qtaub$. Correspondingly, $\bbeta_0$ is the parameter indexing the IDR that achieves this $V_{opt}$. \[thm: limit\_V\] Suppose conditions (C1) is satisfied. We have $ \widehat{V}_{n} = V_{opt} + o_p(n^{-1/2 + \gamma_0}), $ for an arbitrary $\gamma_0 > 0$. *Remark 5.* It is worth emphasizing that Theorem \[thm: limit\_V\] requires much weaker conditions than Theorem \[thm: asymptotic\_dis\_of\_beta\_n\] does. In particular, it does not require the optimal IDR to be unique at the population or sample level. This corresponds to a well known challenging situation where there exists a subpopulation who responds similarly to the two treatment options. If one is willing to assume unique optimal IDR, then the above result can be strengthened to parametric convergence rate, i.e., ${n}^{-1/2}$ rate. Smoothed resampling inference {#smoothb} ----------------------------- Statistical inference for $\widetilde{\bbeta}_{0}$ is challenging due to the nonstandard asymptotic distribution. A natural idea is to use bootstrap. However, the standard nonparametric bootstrap procedure is generally inconsistent for cube-root $M$-estimators (e.g., @Abrevaya2005, @leger2006bootstrap) even for the relatively simpler setting without nuisance functions. As a remedy, $m$-out-of-$n$ bootstrap [@bickel2012resampling], which draws subsamples of size $m$ from the original sample of size $n$ with replacement, has been shown to be consistent for $M$-estimators with a cube root convergence rate in some settings [@delgado2001subsampling]. Theoretically, $m$ depends on $n$, tends to infinity with $n$, and satisfies $m =o(n)$. Practically, choosing an optimal $m$ is not a simple task. Several data-driven approaches for selecting $m$ were investigated but require intensive computation (e.g.,@banerjee2007, @bickel2008choice, @chakraborty2013, @qian2021sequential). In this subsection, we consider an alternative smoothed resampling-based procedure which is computationally more convenient. This approach is motivated by the alternative expression of $\bbeta_0$ (see the derivation of Lemma 1 in the online supplement), given by $$\bbeta_0 = \underset{\bbeta \in \mathbb{B}^o}{\arg\max} ~P g(\cdot, \bbeta, V_{opt}, \Gtrue),$$ where $V_{opt}$ is the optimal value and $g\big(\cdot, \bbeta, v, \Gtrue\big)$ is defined in (\[shark\]). That is, $\bbeta_0$ is the parameter indexing the IDR that achieves the optimal value $V_{opt}$. This naturally leads to an alternative representation of $\widehat{\bbeta}_n$, given by $ \widehat{\bbeta}_n = \underset{\bbeta \in \mathbb{B}^o}{\arg\max} ~n^{-1} \sum_{i=1}^n g\left(\cdot, \bbeta, \widehat{V}_{n}, \Ghat\right). $ To implement the smoothed resampling-based inference, we first obtain the estimator $\widehat{\bbeta}_n$ and then estimate the optimal value function by $ \widehat{V}_{n} = \arg\min\limits _{\substack{b}} \sum_{i=1}^{n} \frac{\ R_{i}\big( \widehat{\bbeta}_n \big)\ \rho_{\tau}\big(Y_{i} - b\big) } {0.5 \Ghat\big(Y_i\big)}. $ Motivated by @wu2021resampling for mean-optimal treatment regime with complete data, we consider the following smoothed estimator $$\overline{\bbeta}_n= \underset{\bbeta \in \mathbb{B}^o}{\arg\max} ~ \frac{1}{n} \sum_{i=1}^{n} \left(2 A_{i}-1\right)\frac{\Delta_{i} I\left(Y_{i}>\widehat{V}_{n}\right)}{G_{C}\left(Y_{i}\right)} K\left(\frac{\vX_{i}^T \bbeta}{h_n}\right),$$ where $K(\cdot)$ is a kernel function and $h_n$ is a bandwidth. The kernel function $K(\cdot)$ is only required to satisfy some general conditions, for example, we can take it to be the cumulative distribution function of the standard normal distribution. Replacing the indicator function in the treatment regime by the kernel function helps alleviates the sharp edge effect. Write $\overline{\bbeta}_n = (\overline{\beta}_{n1}, \overline{\widetilde{\bbeta}}_n^T)^T$ Similarly as in @wu2021resampling, it is expected that $\sqrt{nh}\left(\overline{\widetilde{\bbeta}}-\widetilde{\beta}_{0}\right)$ is asymptotically normal. Note that $\overline{\bbeta}_n$ minimizes the loss function $-n^{-1} \sum_{i=1}^{n} \left(2 A_{i}-1\right)\frac{\Delta_{i} I\left(Y_{i}>\widehat{V}_{n}\right)}{G_{C}\left(Y_{i}\right)} K\left(\frac{\vX_{i}^T \bbeta}{h_n}\right)$, which is a smoothed estimator of a weighted mis-classification error. We choose $h_n$ by five-fold cross-validation based on this loss function. The asymptotic covariance matrix is complex and involves unknown counter-factual distributions. For inference, we consider the following perturbed smoothed estimator $$\overline{\bbeta}_n^*= \underset{\bbeta \in \mathbb{B}^o}{\arg\max} ~ \frac{1}{n} \sum_{i=1}^{n} \xi_i\left(2 A_{i}-1\right)\frac{\Delta_{i} I\left(Y_{i}>\widehat{V}_{n}\right)}{G_{C}\left(Y_{i}\right)} K\left(\frac{\vX_{i}^T \bbeta}{h_n}\right),$$ where $\xi_{1}, \cdots, \xi_{n}$ are positive random weights independent of the data, with mean one and variance one. To obtain the bootstrap distribution of $\overline{\bbeta}_n^*$, we repeatedly generate independent random weights and solve for the smoothed estimator. Write $\overline{\bbeta}_n = (\overline{\beta}_{n1}, \overline{\beta}_{n2}, \ldots, \overline{\beta}_{np})$, and $\overline{\bbeta}_n^* = (\overline{\beta}_{n1}^*, \overline{\beta}_{n2}^*, \ldots, \overline{\beta}_{np}^*)$, where $|\overline{\beta}_{n1}^*|=1$. For $j=2,\ldots,p$, let $\eta_j^{*(\alpha/2)}$ and $\eta_j^{*(1-\alpha/2)}$ be the $(\alpha/2)$-th and $(1-\alpha/2)$-th quantile of the bootstrap distribution of $(nh_n)^{1/2}(\overline{\beta}_{nj}^*-\overline{\beta}_{nj})$, respectively, where $\alpha$ is a small positive number. We can estimate $\eta_j^{*(\alpha/2)}$ and $\eta_j^{*(1-\alpha/2)}$ from a large number of bootstrap samples. An asymptotic $100(1-\alpha)\%$ bootstrap confidence interval for $\beta_{0j}$, $j=2,\ldots, p$, is given by $ \big\{\overline{\beta}_{nj}-(nh_n)^{-1/2}\eta_j^{*(1-\alpha/2)}, \overline{\beta}_{nj}-(nh_n)^{-1/2}\eta_j^{*(\alpha/2)}\big\}. $. Estimation of optimal dynamic IDR with censored data. ===================================================== In this section, we consider the extension to the dynamic decision problem which involves multiple decision points. The decision at a later stage can depend on baseline covariates, treatment history, and intermediate variables such as how the subject responds to earlier treatment(s). For survival data, complications arise as the subject may be censored anytime before the end of the study, which results in an incomplete trajectory of treatments. Potential outcome framework --------------------------- For clarity, we focus on a two-stage dynamic decision problem with random right censoring. We consider a setup similar as @jiang2017estimation but will define the potential outcome more carefully. At the beginning of a study (time point 0), baseline covariates $\vX_1$ of patients would be collected, and each of them is assigned one of two first-stage treatment options, say $A_1$ and $A_2$. Then the second stage starts from a prespecified time $s$ with $s>0$. Additional intermediate covariates $\vX_2$ reflecting the reaction to first stage treatment up to time $s$ would be collected if applicable, and those subjects who remain at risk at time $s$ is assigned one of two first-stage treatment options, say $B_1$ and $B_2$. $\{B_1,B_2\}$ may not overlap with $\{A_1,A_2\}$. For example, in making decisions for cancer patients, $\{A_1,A_2\}$ could be induction treatments, while $\{B_1,B_2\}$ may represent maintenance treatment or salvage treatment. Similarly as in the one-stage setting, the potential survival time is defined when censoring is absent, and we would like to estimate the optimal dynamic IDR with a criterion based on this potential survival time when the real data is complicated by censoring. Let $D_i$ denote the random treatment at stage $i$ when the subject is eligible. Note that $D_2$ may not exist if the patient is not at risk at time $s$. Consider a sequential IDR $\bm{d} = (d_1,d_2)$, where $d_1(\vX_1) \in \{A_1,A_2\}$ and $d_2\left( \vX_1, D_1, \vX_2 \right) \in \{B_1,B_2\}$. A subject is considered to be consistent with $\bm{d}$ if he/she receives a first treatment $D_1$ that equals $d_1(\vX_1)$ and a second treatment $D_2$ which equals $d_2(\vX_1, D_1, \vX_2)$ (full compliance); *or* receives treatment $D_1$ complying with the rule $d_1$ at stage one but does not survive long enough to be eligible for stage two treatment. Let $ \dot{R}(d_1)= I\Big({{\dot{T}\left(d_1, \emptyset\right)}}> s\Big) $ indicate the subject's eligibility status for stage two treatment when complying with rule $d_1$, where ${{\dot{T}\left(d_1, \emptyset\right)}}$ is shorthand notation of $\dot{T}(d_1(\vX_1), \emptyset)$, which represents the potential survival time if the subject receives $d_1$ without stage-two action. Let ${{\dot{T}\left(d_1, d_2\right)}}$ be the potential survival time if the subject receives the full sequence of treatments $(d_1, d_2)$. Implicitly, ${{\dot{T}\left(d_1, d_2\right)}}> s$. Let $T^{\ast}(\bm{d})$ be the potential survival time if the subject is consistent with the treatment sequence $\bm{d}$. We can write $$\label{eq: po_dyn} T^{\ast}(\bm{d}) = {{\dot{T}\left(d_1, \emptyset\right)}}\big(1 - \dot{R}(d_1)\big) + {{\dot{T}\left(d_1, d_2\right)}}\dot{R}(d_1).$$ We are interested in estimating the optimal sequential decision $\bm{d} = (d_1,d_2)$ in some class $\mathcal{D}$, that is, $\bm{d}_{opt} = \arg\max_{\bm{d}\in\mathcal{D}} Q_{\tau}(\bm{d})$. Define $H_1 = \{\vX_1 \}$, and define $H_2^{\ast}(d_1) = \left\{ \vX_1, d_1, \vX_2^{\ast}(d_1) \right\}$, where $\vX_2^{\ast}(d_1)$ denotes the potential intermediate information between decision 1 and decision 2 had the subject started with treatment $d_1(\vX_1)$ and given $\dot{R}(d_1)=1$. Denote the set of potential outcomes as $$O^{\ast}(\bm{d}) = \left\{{{\dot{T}\left(d_1, \emptyset\right)}}, \ \dot{R}(d_1), \ \dot{R}(d_1)\vX_2^{\ast}(d_1),\ \dot{R}(d_1) d_2\big(H_2^{\ast} (d_1) \big), \ T^{\ast} (\bm{d})\right\}.$$ Robust estimation of optimal IDR -------------------------------- When censoring is absent, for a given subjects, we would observe the first stage treatment $D_1$ and the corresponding $\dot{R}(D_1)$. The consistency assumption for causal inference, similar as in the one-stage setting, ensures that the observed survival time $T$ would satisfy $$\label{T_censored_data} T = \begin{cases} \sum_{ j\in\{1,2\} } I\{D_1=A_j\} \dot{T}(A_j, \emptyset), & \mbox{if } \dot{R} = 0 \\ \sum_{j\in\{1,2\},\ k\in\{1,2\} } I\{D_1 =A_j, D_2 =B_k\} \dot{T}\big(A_j, B_k\big) & \mbox{if } \dot{R} = 1. \end{cases}$$ Due to censoring, we may not observe $T$. Denote the actually observed survival time under possible censoring as $Y = \min\{T,C\}$, and let $\Delta = I(T \leq C)$ be the censoring indicator. Further, let $\Gamma = I(C > s)$ denote whether censoring occurs in the first stage. As a result of censoring, only those subjects that survived longer than $s$ and are not censored before $s$ are eligible for the second-stage treatment, for whom the trajectory observed up to time $s$ is $H_{2} = \big\{\vX_{1}, D_1, \vX_{2} \big\}$. When the trial ends, the observed data is $$\label{observed_data_model} \left\{ \vX_{i1}, D_1,\ \dot{R}_i,\ \Gamma_i, \ \dot{R}_i \Gamma_i \vX_{i2},\ \dot{R}_i \Gamma_i D_2,\ Y_i,\ \Delta_i \right\}, \mbox{ for } i=1,\ldots,n.$$ Based on the observed data, our goal is to estimate the optimal sequential decision rule within a given class $\mathcal{D}$. Extending the one-stage formulation, we consider sequential IDR of the form $ \bm{d}_{\bm{\xi}}=\big\{ d_{1,\bbeta}(H_1) , d_{2,\bm{\zeta}}(H_2)\big\}$, where $ d_{1,\bbeta}(H_1) = d_{1,\bbeta}(\vX_1) = I \big(\vX_1^T\bbeta > 0\big)$, $d_{2,\bm{\zeta}}(H_2) = d_{2,\bm{\zeta}}(\vX_1, D_1, \vX_2) = I \big( H_2^T\bm{\zeta} > 0\big)$ and $\bm{\xi}=(\bbeta^T, \bm{\zeta}^T)^T$. Without loss of generality, we assume that if $d_{1,\bbeta}(\vX_1) = 0$, then the recommended first-stage treatment is $A_1$, otherwise is $A_2$; and if $d_{2,\bm{\zeta}}(H_2) = 0$, then the recommended second-stage treatment is $B_1$, otherwise is $B_2$. For identifiability, we assume $\bbeta$ and $\bm{\zeta}$ satisfy $\vert \beta_1 \vert=1$ and $\vert \zeta_1 \vert=1$. Hence $\mathcal{D} = \big\lbrace \bm{d}_{\bm{\xi}} : \bm{\xi}\in \widetilde{\mathbb{C}} \big\rbrace$ where $\widetilde{\mathbb{C}} = \{-1,1\}\times \widetilde{\mathbb{B}}\times \{-1,1\}\times \widetilde{\mathbb{Z}} $, with $\widetilde{\mathbb{B}}$ being a compact subset of $\mathbb{R}^{p_1 - 1}$, $\widetilde{\mathbb{Z}}$ being a compact subset of $\mathbb{R}^{p_2 - 1}$; $p_1$ is the dimension of $\vX_1$, and $p_2$ is the dimension of $(\vX_1^T, D_1, \vX_2^T)^T$. The parameter $\bm{\xi}_0$ indexing the optimal dynamic IDR in $\mathcal{D} $ is defined by: $$\bm{\xi}_0 = \underset{ \bm{\xi}\in \widetilde{\mathbb{C}} }{\arg\max\ } Q_{\tau}\big\{T^{\ast}\big( \bm{d}_{\bm{\xi}}\big)\big\},$$ where $Q_{\tau} \lbrace \cdot \rbrace $ denotes the marginal $\tau$th quantile ($0<\tau<1$), and $T^{\ast}(\bm{d}_{\bm{\xi}})$ is obtained by setting $\bm{d} = \bm{d}_{\bm{\xi}}$ in (\[eq: po\_dyn\]). To extend the policy-search method to estimate $\bm{\xi}_0$, we define $$\widetilde{R}( \bm{d}_{\bm{\xi}} ) = \Delta I\big(D_{1} = d_{1,\bbeta}(\vX_{1})\big) \ \big[ I (Y \leq s) + I(Y>s) I\big(D_{2} = d_{2,\bm{\zeta}}(H_{2})\big) \big].$$ For subjects with $\widetilde{R}( \bm{d}_{\bm{\xi}} ) = 1$, we observe the potential survival time of interest, that is, $Y = T^{\ast}(\bm{d}_{\bm{\xi}})$. For simplicity in presentation, we assumed that the data are from a sequential multiple assignment randomized trial (SMART, [@LD00; @Murphy08]), where the randomization probabilities at each stage are known by design. That is, at stage one, $P(D_1 = A_2) = 1- P(D_1 = A_1) = \pi_1$, while at stage two $P(D_2 = B_2|Y_i >s, C_i > s) = 1-P(D_2=B_1|Y_i>s, C_i > s) = \pi_2$. Given $\bm{d} = (d_1, d_2) \in \mathcal{D}$, let $ \pi_{d_1}(\vX_{i1}) = \pi_1 d_1(\vX_{i1}) + (1-\pi_1)\{1-d_1(\vX_{i1})\} $ denote the probability of compliance to $d_1$ at stage one; and let $\pi_{d_2}(H_{i2}) = \pi_2 d_2(H_{i2}) + (1-\pi_2)\{1-d_2(H_{i2})\}$ denote the probability of compliance to $d_2$ at stage two, given that stage one's target potential data is observed and also $Y>s, C>s$. Overall, the probability to observe $T^{\ast}(\bm{d})$ is $$\label{eq: lion} \widetilde{w}_{\bm{d},i} = \ P\left(\widetilde{R}_i(\bm{d})=1 \mid \vX_{i1}, O^{\ast}_i\big(\bm{d}\big) \right) = \pi_{d_1}(\vX_{i1})\Gtrue (Y_i) \big\{ I(Y_i \leq s) + \pi_{d_2}(H_{i2})I (Y_i > s) \big\}.$$ We estimate $ Q_{\tau}\big\{T^{\ast}(\bm{d})\big\} $ by $$\widehat{Q}_{\tau}\big(T^{\ast}(\bm{d}); \Ghat \big) = \underset{b}{\arg\min} \sum_{i=1}^{n} \frac{\ \widetilde{R}_i( \bm{d}) \rho_{\tau}\big(Y_{i} - b\big) } { \widehat{w}_{\bm{d},i}},$$ where $\widehat{w}^{(2)}_{\bm{d},i}$ is obtained by plugging in the Kaplan-Meier estimator for $G_C$ in (\[eq: lion\]). For brevity, we use the shorthand notation $\widehat{Q}_{\tau}\big(\bm{\xi}; \Ghat \big)$ for $\widehat{Q}_{\tau}\big(T^{\ast}(\bm{d}_{\bm{\xi}}); \Ghat \big)$. Let $L$ denote the end of the study. Assume there exists a constant $\eta > 0$ such that $\Gtrue(L) > \eta > 0$. Furthermore, assume $C$ has a continuously differentiable density function which is bounded away from infinity on $(0, L)$. Also, $m'_0 \triangleq \sup_{\bm{\xi} \in \widetilde{\mathbb{C}}} Q_{\tau} \{T^{\ast}(\bm{d}_{\bm{\xi}})\} < L$. Consider an arbitrary treatment sequence $\bm{d} = (d_1 , d_2)$, with $d_1(H_1) \in \{A_1, A_2\}$ and $d_2(H_2) \in \{B_1, B_2\}$. Marginally, ${{\dot{T}\left(d_1, \emptyset\right)}}$ and ${{\dot{T}\left(d_1, d_2\right)}}$ have continuous distributions with continuously differentiable density functions. $\forall \bm{\xi} \in \widetilde{\mathbb{C}}$, let $f_{T^{\ast}(\bm{d}_{\bm{\xi}})}(\cdot)$ denote the marginal density function of the distribution of the potential survival time $T^{\ast}(d_{\bm{\xi}})$. There exist positive constants $\kappa_1$ and $\delta$, such that $\inf_{\bm{\xi} \in \widetilde{\mathbb{C}}} \ \inf_{|m - m'_0| \leq \delta} f_{T^{\ast}(d_{\bm{\xi}})}(m)\geq\kappa_1$. The following lemma states the consistency of $\widehat{Q}_{\tau}\big(\bm{\xi}; \Ghat \big)$ for the marginal quantile of $ T^{\ast}(\bm{d}_{\bm{\xi}})$. \[lemma: consistency\_DTR\] For all $ d_{\bm{\xi}} \in \mathcal{D} $, we have $ \widehat{Q}_{\tau}\big( \bm{\xi}; \Ghat \big) \rightarrow Q_{\tau}\{ T^{\ast}(\bm{d}_{\bm{\xi}}) \} $ in probability. Hence, an estimator of the parameter $\bm{\xi}_0$ is $$\widehat{\bm{\xi}}_n = \underset{ \bm{\xi}\in \widetilde{\mathbb{C}} }{\arg\max\ } \widehat{Q}_{\tau}\big\{T^{\ast}\big( \bm{d}_{\bm{\xi}}\big); \Ghat \big\}.$$ The optimal value function, the maximally achievable marginal $\tau$-th quantile of the potential outcome distribution considering all IDRs in $\mathcal{D}$, is given by $ V_{opt} = \underset{ \bm{\xi}\in \widetilde{\mathbb{C}} }{\sup\ } Q_{\tau}\big\{T^{\ast}(d_{\bm{\xi}})\big\} . $ An estimator of this quantity is $\widehat{V}_n = \ \widehat{Q}_{\tau}\big( \widehat{\bm{\xi}}_n, \Ghat \big)$. Similarly as the one-stage setting, we can re-express $\widehat{V}_n$ as $$\label{eq: Vn_dyn} \widehat{V}_n = \sup \left\{m: \underset{ \bm{\xi}\in \widetilde{\mathbb{C}} }{\sup\ } n^{-1} \sum_{i=1}^{n} g\left(\cdot, \bm{\xi},m,\Ghat \right) \right\}$$ where $ g\left(\cdot, \bm{\xi},m,\Gtrue \right) = \frac{\widetilde{R}( \bm{d}) I(Y-m > 0)}{ \widetilde{w}_{\bm{d}}}. $ The following theorem shows that in the dynamic setting, the optimal value function can be estimated with a near parametric rate. \[thm: limit\_V\_dyn\] For the estimator $\widehat{V}_{n}$ defined in (\[eq: Vn\_dyn\]), we have $ \widehat{V}_{n} = V_{opt} + o_p(n^{-1/2 + \gamma_0}) $ for an arbitrary $\gamma_0 > 0$. It is worth noting that the above result does not require the optimal IDR to be unique. Similar nonregular asymptotic distribution for $\widehat{\bm{\xi}}_n$ can also be established using the same idea as in Section \[sec: statistical properties of single-stage estimator\] but more complex notations.\ *Remark 6.* The method we propose for the dynamic setting is different from the Q-learning approach, which searches for optimal treatment regimes starting from the last stage and moving backward. The Q-learning approach was extended to censored data by @goldberg2012q. There exist several distinct differences between these two methods. First, the proposed method is model-free in the sense that it does not require to specify an outcome regression model. The Q-learning approach is model-based and requires a survival time model that incorporates both the covariate effects and treatment-covariate interaction effects. Second, the proposed method considers a quantile-optimal criterion while @goldberg2012q adopts a restricted mean criterion. Finally, from a theoretical perspective, this work focuses on the statistical properties of the estimated parameters indexing the optimal IDR while @goldberg2012q focuses on the finite sample bound of the generalization error of the estimated optimal IDR. Numerical studies ================= Monte Carlo simulations ----------------------- We report simulation results for three different settings. In the first example, we estimate one-stage optimal treatment under random censoring; in the second example, we estimate one-stage optimal treatment under covariate-dependent censoring; while in the third example, we consider a two-stage dynamic optimal IDR estimation problem.\ <span>**Example 1 (random censoring)**</span>. We generate the random sample $ \{\vX_{i},A_{i},Y_{i},\Delta_{i}\} $, $ i=1,2,...,n $, from the model: $X_{1} \sim U(0,1)$, $\TstarZero\big| X_1 \sim \mbox{Weibull}(\mbox{shape} = 1, \mbox{scale} =1) + 1$, $\TstarOne \big| X_1 \sim \mbox{Weibull}(\mbox{shape} = 3, \mbox{scale} =0.5 + X_1) + 2X_1$, $A\big| X,\TstarZero,\TstarOne \sim \mbox{Bernoulli}(0.5)$. The response variable in the absence of censoring is generated by $T = \TstarZero\left(1-A\right) + \TstarOne A$. The censoring time $C$ has a constant density function $0.22$ on $(0,2)$ and a constant density function $ 0.07$ on $[2,10)$. The observed response is $ Y= \min\{T, C \}$ and the censoring indicator is $\Delta = I\left\{ T \leq C \right\}$. This setup achieves an overall censoring rate of 35%. To illustrate the heterogeneous treatment effects, we split $ X_1 $ into two strata: $ \left[0, 0.5\right] $ and $ \left(0.5, 1\right]$. Figure \[fig1\] displays the histograms of $\TstarZero$ and $\TstarOne$ in each stratum. This plot provides strong evidence that $X_1$ has a qualitative interaction with the treatment. Intuitively, the optimal IDR should depend on $X_1$. We will apply the proposed method to estimate the quantile-optimal IDR for $ \tau=0.25 $ and $\tau=0.5$, respectively. We consider the following class of IDRs $ \mathcal{D}=\left\{\dbetax=\mbox{I}\left(X_1\beta_1 + \beta_2 > 0\right): \vert\beta_{1}\vert=1, \beta_2 \in \mathbb{R} \right\}. $ Denote the parameter indexing the $\tau$th quantile optimal IDR in $\mathcal{D}$ by $\bbeta_0^{(\tau)}$. For each $\tau$, we use a large Monte Carlo data set of size $n = 10^7$ to estimate $\bbeta_0^{(\tau)}$ and the $\tau$th quantile of the potential outcome in the above class of IDRs (denoted by $Q_{\tau}$) and treat the results as population parameter values, see Table \[tab: Example1\_truth\]. Consider, for example, the row corresponding to $\tau=0.5$. We apply the 0.5-quantile optimal IDR to assign treatment in a large independent Monte Carlo sample. Assume everyone in the population follows the recommended treatment and records his/her outcome. The median of the potential outcome distribution is 2.258, the first quartile of the potential outcome distribution is 1.587. ![Histograms of $\TstarZero $ and $ \TstarOne $ stratified by $X_1$](hist1.pdf "fig:"){width="10cm"} \[fig1\] We compare the proposed estimator (denoted by New) with the naive estimator (denoted by Naive), which ignores censoring and pretends all observations are complete [@wang2017quantile]. We conduct the simulation experiment with 400 replications for sample size $ n= $ 300, 500, and 1000. In this experiment, we observed that New always correctly estimates the sign of $\beta_{01}$ for both $\tau=0.25$ and $\tau=0.5$, while Naive has 4%, 2%, and 1% error rate for $n=300,500$ and $1000$ respectively in estimating the sign of $\beta_{01}$ for $\tau=0.25$ (0% error rate for $\tau=0.5$). Consideration this phenomenon, we conservatively compare the estimates for $\beta_{02}$ for the two methods in cases where $\hat{\beta}_{01} = 1$. Table \[one\_cov\_RandomCen\] summarizes the bias (with standard deviation in the parenthesis) of New and Naive for estimating $\bbeta_0^{(\tau)}$ and $Q_{\tau}$ for different combinations of $\tau$ and $n$. To estimate $Q_{\tau}$, New plugs $\widehat{\bbeta}_n$ into the formula of $\widehat{Q}_{\tau}\left(\cdot; \Ghat\right)$ in (\[eq:QtauEstimator\_Ghat2\]); and Naive does plugs-in similarly pretending all observation were complete. We observe that New has satisfactory performance, while Naive exhibits substantial bias for estimating both $\bbeta_0^{(\tau)}$ and $Q_{\tau}$. $\tau$ $ {\beta}_{01}^{(\tau)} $ $ {\beta}_{02}^{(\tau)} $ $ {Q}_{0.25}$ $ {Q}_{0.5}$ -------- --------------------------- --------------------------- --------------- -------------- 0.25 1 -0.428 1.658 2.215 0.50 1 -0.552 1.587 2.258 : Example 1: Parameters indexing the quantile-optimal IDRs ($\tau=0.25$ and $0.5$) in $\mathcal{D}$ and the $\tau$th quantile of the potential outcome distribution (denoted by $Q_{\tau}$), based on a Monte Carlo experiment ($n = 10^7$). \[tab: Example1\_truth\] <span>cr cc c cc</span> $\tau$ & $ n $ & &&\ && $\beta_{02}^{(\tau)}$ & $Q_{\tau}$ && $\beta_{02}^{(\tau)}$ & $Q_{\tau}$\ \ & 300 & 0.005(0.066) & 0.056(0.113) & & -0.025(0.204) & -0.555(0.057)\ & 500 & -0.001(0.054) & 0.027(0.082) & & -0.043(0.213) & -0.568(0.050)\ & 1000 & 0.001(0.043) & 0.020(0.055) & & -0.048(0.202) & -0.585(0.031)\ & 300 & 0.002(0.098) & 0.048(0.124) & & 0.129(0.082) & -0.618(0.078)\ & 500 & 0.003(0.080) & 0.023(0.101) & & 0.122(0.064) & -0.617(0.065)\ & 1000 & -0.002(0.051) & 0.022(0.061) & & 0.134(0.051) & -0.649(0.049)\ \[one\_cov\_RandomCen\] Finally, we demonstrate the smoothed resampling procedure in Section \[smoothb\] for inference. We consider 90% and 95% confidence intervals for $\beta_{02}$ for $\tau=0.25$ and 0.5, respectively. The empirical coverage probabilities and average confidence interval lengths are reported in Table \[inferenceTab\] for $n=1000$ based on 400 bootstrap samples. The observed empirical coverage probabilities are close to the nominal levels with reasonable lengths. ------ --------------- ---------- -------- -- ---------- -------- -- -- -- -- -- -- -- -- -- -- $n$ $\tau$ coverage length coverage length 500 $\tau =0.5 $ 0.89 0.17 0.92 0.21 $\tau =0.25 $ 0.91 0.29 0.95 0.34 1000 $\tau =0.5 $ 0.88 0.13 0.93 0.16 $\tau =0.25 $ 0.88 0.14 0.93 0.16 ------ --------------- ---------- -------- -- ---------- -------- -- -- -- -- -- -- -- -- -- -- : Confidence intervals for $\beta_{02}$ using smoothed resampling \[inferenceTab\] <span>**Example 2 (covariate-dependent censoring)**</span>. Let $\vX_{i}=\left(X_{i1},1,X_{i2}\right)^{T}$, where $X_{i1}$, $X_{i2}$ are independent $\mbox{Uniform}\left(0,1\right)$ random variables. The binary treatment $A_i$ is independent of $\vX_{i}$ and satisfies $P(A_i = 1) = 0.5$. The distribution of censoring variable $C_{i}$ is $$C_{i}=\begin{cases} 4+\left(2-X_{i1}\right)\omega_{i}, & \mbox{if }A_{i}=0\\ 2+I\left(X_{i1}<0.5 \cup X_{i2}<0.5 \right)+\omega_{i}, & \mbox{if }A_{i}=1 \end{cases},$$ where the $\omega_{i}$'s are independent $N(0,1)$ random variables. The survival time $T_i$ is generated by $$T_{i}=1+X_{i1}+X_{i2}+A_i\left(3-3X_{i1} - 1.5X_{i2}\right)+\left[0.5+A_{i}\left(1+X_{i1}+X_{i2}\right)\right]\epsilon_{i},\label{eq:surv-model}$$ where the $\epsilon_{i}$'s are independent normal random variables with mean zero and standard deviation 0.5. The observed response is $Y_i=\min\left\{T_i, C_i\right\}$. This configuration yields a 30% censoring rate. We consider estimating the $\tau$th quantile optimal IDR ($\tau$=0.1 and 0.25) within the class $\mathbb{D}=\big\{ \mbox{I}\left(\beta_1 X_1 + \beta_2 + \beta_3 X_2 > 0\right): \vert \beta_1\vert =1, \left(\beta_2, \beta_3\right)^{T}\in {\mathbb{R}^2}\big\}$. Similarly as for example 1, the parameters indexing the quantile-optimal IDRs ($\tau=0.1$ and $0.25$) and the the maximally achievable $\tau$th quantile of the potential outcome (denoted by $Q_{\tau}$) in $\mathbb{D}$ were estimated based on a large Monte Carlo experiment with sample size $n = 10^7$ and treated as population parameter values, see Table \[tab\_Optimal-Treatment-Regimes\_MC\]. To incorporate covariate-dependent censoring, we adopt the local Kaplan-Meier estimator (with bandwidth $h_n = 0.08, 0.1, 0.12, 0.14$) described in Remark 1 of Section 2 to estimate the propensity score. Table \[tab\_Example2\_0.1\] summarizes the simulation results for New and Naive for $n=500$ based on 300 replications. We observe that New has satisfactory performance and its performance is stable with respect to different choices of the bandwidth $h$. In contrast, Naive exhibits substantial bias for estimating $\bbeta_{02}^{(\tau)}$ and $Q_{\tau}$.\ $\tau$ $\beta_{01}^{(\tau)}$ $\beta_{02}^{(\tau)}$ $\beta_{03}^{(\tau)}$ $Q_{\tau}({\bm{\beta} })$ -------- ----------------------- ----------------------- ----------------------- --------------------------- 0.10 -1 0.896 -0.774 1.853 0.25 -1 1.140 -0.825 2.247 : Parameters indexing the quantile-optimal IDRs ($\tau=0.1$ and $0.25$) and the the maximally achievable $\tau$th quantile of the potential outcome (denoted by $Q_{\tau}$) in $\mathbb{D}$ in Example 1, based on a Monte Carlo experiment ($n = 10^7$). \[tab\_Optimal-Treatment-Regimes\_MC\] \[tab\_Example2\_0.1\] <span>**Example 3 (Two-stage dynamic individualized decision rule).**</span> The simulation setup is motivated by the example in @jiang2017estimation. The censoring time $ C \sim \mbox{Unif}\left(0, C_0\right) $, where $C_0$ is a positive constant. Let $s=1$. Generate the data up to time $s$, $\big\lbrace X_1,D_1, \mathsf{Z}^C= \mbox{I}\big(\min\big(T_1, C\big) >s \big) \big\rbrace,$ from the following distributions: $X_1 \sim \mbox{Unif}\left(0, 4\right)$, $D_1\mid X_1 \sim \mbox{Bernoulli}(0.5)$ and $T_1\mid X_1, D_1 \sim \mbox{Exp}\big(\lambda_1 ( X_1, D_1) \big)$ where $\lambda_1(\cdot)$ is a rate function to be specified later, and $\mathsf{Z}^C$ is an auxiliary variable that equals the product of $\dot{R}$ and $\Gamma$ in observed data model (\[observed\_data\_model\]). If $\mathsf{Z}^C = 1$, then the simulated patient is eligible for stage-two treatment. We generate the intermediate covariate $X_2$, second stage treatment $D_2$ and time $T_2$, representing the survival time after time $s$ according to: $ e \sim \mbox{Unif}\left(0, 2\right)$, $ X_2\mid X_1, D_1 = 0.5X_1 - 0.4\big(D_1 - 0.5\big) + e$, $ D_2\mid X_1, D_1, X_2 \sim \mbox{Bernoulli}(0.5) $, $ T_2\mid X_1, D_1, X_2, D_2 \sim \mbox{Exp}\big(\lambda_2 (X_1, D_1, X_2, D_2) \big)$, where $\lambda_2(\cdot)$ is a rate function to be specified later. For $\mathsf{Z}^C = 1$, the observed survival time and censoring status are $Y = \min\{(s+T_2), C\}$ and $\Delta = \mbox{I}\{(s+ T_2) \leq C\}$, respectively; for $\mathsf{Z}^C = 0$, the observed survival time and censoring status are $Y=\min\{T_1, C\}$ and $\Delta = \mbox{I}\{T_1 \leq C\}$, respectively. Let $H_1 = \{X_1\}$ and $H_2 = \{X_1, D_1, X_2\}$. For rate functions for $T_1$ and $T_2$, consider three scenarios: 1. $\lambda_1(H_1, D_1) = 0.5\exp\big(1.75(D_1-0.5)(X_1-2)\big)$,\ $\lambda_2(H_2, D_2) = 0.3\exp\big(2.5(D_2-0.4)(X_2-2) - D_1(X_1-2)\big)$ 2. $\lambda_1(H_1, D_1) = 0.2\exp\big(2(D_1-0.5)(-X_1 + 2)\big)$,\ $\lambda_2(H_2, D_2) = 0.2\exp\big(1.5 (D_2 - 0.5)(X_2 - 2) + 0.3X_1 + 0.3X_2\big)$ 3. $\lambda_1(H_1, D_1)= 0.3\exp\big(3(D_1-0.3)(X_1-3)\big)$,\ $\lambda_2(H_2, D_2)=0.3\exp \big(2(D_2 - 0.5)(X_2-2) - 0.5(D_1-0.3)(X_1 -3)\big)$ For each scenario, we consider two different choice of $C_0$ to achieve 15% and 40% overall censoring rate, respectively. In this setup, $T_1$ serves as the underlying $\sum_{ j\in\{1,2\} } \mbox{I}\{D_1=A_j\} \dot{T}(A_j, \emptyset) $ in equation (\[T\_censored\_data\]); $T_2$ is the survival time after initiation of the second stage treatment conditional on $\mathsf{Z}^C=1$, hence $T_2 + s$ serves as $\sum_{j\in\{1,2\},\ k\in\{1,2\} } \mbox{I}\{D_1 =A_j, D_2 =B_k\} \dot{T}\big(A_j, B_k\big)$ in (\[T\_censored\_data\]). By the interaction between $D_2$ and $H_2$ in $\lambda_2 (\cdot)$ functions, the three scenarios share the same true optimal second-stage strategy for patients with $\mathsf{Z}^C=1$, which is $d_{2}^{opt}(H_2) = \mbox{I}(-X_2+2 > 0)$. Suppose the class of IDRs is $\mathcal{D} = \big\lbrace \bm{d}_{\bm{\xi}} =(d_{1,\bbeta}, d_{2,\bm{\zeta}}): d_{1,\bbeta}(X_1) =\mbox{I}\{\beta_{1}X_1 + \beta_2 > 0\}, d_{2,\bm{\zeta}}(X_2) =\mbox{I}\{\zeta_{1}X_2 + \zeta_2 > 0\}, \vert \beta_{1} \vert = 1, \vert \zeta_{1} \vert = 1 \big\rbrace$. Thus, $d_{2}^{opt}(H_2)$ is contained in $\mathcal{D} $, and the parameter indexing $d_{2}^{opt}(H_2)$ in $\mathcal{D} $ is ${\zeta_{01} =-1,\zeta_{02} =2 }$. However, for all three cases, the true value for $\bbeta_0$ indexing the optimal first-stage treatment in $\mathcal{D} $ does not have close form representations. We used grid-search with sample size $n=10^7$ for $\bbeta \in \{-1\}\times [0,4]\cup \{1\}\times [-4,0]$ to obtain $\bbeta_0$ in these cases, where the search space is the largest set of identifiable $\bbeta$ since $X_1$ has support $[0,4]$. With the above setup, we generate a random sample $\{X_{i1}, D_{i1}, Z^C_{i} X_{i2}, Z^C_{i}D_{i2}, Y_i, \Delta_i \}$, $i=1,\ldots,n$, where $n=300,500$ and 1000 are considered. We assume the randomization probability $\pi_1$ and $\pi_2$ are known, and applied the proposed method to estimate the optimal IDR with $\tau=0.3$. Table \[DTR\_results\] reports the simulation estimates of bias and standard deviations of the parameter indexing the quantile optimal dynamic regime $\widehat{\bm{\xi}}_n$. It also reports estimates of bias and standard deviation of the plug-in estimator of maximal achievable 0.3-quantile, $\widehat{Q}_{0.3}\{ T^{\ast}(\bm{d}_{\widehat{\bm{\xi}}_n}) \}$. We observe the proposed method reliably estimated the optimal two-stage IDR. The average biases and standard deviations of $\hat{\beta}_{02}$ and $\hat{\zeta}_{02}$ decrease as sample size increases. The lower censoring rate corresponds to better performance. Case n C% --------------------- ------ ------ ---- --------------- --------------- -------------- 300 40 -0.052(0.499) 0.035(0.391) 0.525(0.445) 500 40 -0.036(0.416) 0.031(0.339) 0.353(0.358) 1000 40 -0.005(0.317) 0.043(0.282) 0.206(0.220) (r)<span>3-7</span> 300 15 0.016(0.402) -0.008(0.356) 0.331(0.345) 500 15 0.010(0.329) 0.006(0.303) 0.247(0.262) 1000 15 -0.008(0.255) 0.001(0.230) 0.139(0.187) 300 40 -0.030(0.648) -0.023(0.371) 0.280(0.225) 500 40 -0.031(0.581) -0.015(0.344) 0.181(0.149) 1000 40 -0.016(0.429) -0.013(0.279) 0.120(0.105)
{'source': 'arXiv_out/2204'}