Deep Reinforcement Learning for Dynamic
Treatment Regimes on Medical Registry Data
Ning Liu1,+ , Ying Liu2,*+ , Brent Logan2 , Zhiyuan Xu1 , Jian Tang1 , and Yanzhi Wang1
1 Department

of Electrical Engineering and Computer Science, Syracuse University, Syracuse, NY 13244, USA
of Biostatistics, Medical College of Wisconsin, Milwaukee, WI 53226, USA
* corresponding author email: summeryingl@gmail.com
+ these authors contributed equally to this work

arXiv:1801.09271v1 [] 28 Jan 2018

2 Division

ABSTRACT
This paper presents the first deep reinforcement learning (DRL) framework to estimate the optimal Dynamic Treatment
Regimes from observational medical data. This framework is more flexible and adaptive for high dimensional action and state
spaces than existing reinforcement learning methods to model real-life complexity in heterogeneous disease progression and
treatment choices, with the goal of providing doctor and patients the data-driven personalized decision recommendations. The
proposed DRL framework comprises (i) a supervised learning step to predict the most possible expert actions, and (ii) a deep
reinforcement learning step to estimate the long-term value function of Dynamic Treatment Regimes. Both steps depend on
deep neural networks. As a key motivational example, we have implemented the proposed framework on a data set from
the Center for International Bone Marrow Transplant Research (CIBMTR) registry database, focusing on the sequence of
prevention and treatments for acute and chronic graft versus host disease after transplantation. In the experimental results, we
have demonstrated promising accuracy in predicting human experts’ decisions, as well as the high expected reward function in
the DRL-based dynamic treatment regimes.

Introduction
Medical treatments often compose a sequence of intervention decisions that are made adaptive to the time-varying clinical status
and conditions of a patient, which are coined as Dynamic Treatment Regimes (DTRs1 ). “How can we optimize the sequence of
specific treatments for specific patients?” is a central question of precision medicine. More specifically, the scientific question
our paper focuses on is the determination of the optimal DTRs to maximize the long-term clinical outcome.
When the straightforward rule-based treatment guidelines are difficult to be established, the statistical learning method
provides a data-driven tool to explore and examine the best strategies. These data driven approaches leverage on the technology
advances to collect the increasingly abundant medical data (e.g., clinical assessments, genomic data, electronic health records)
from each individual patient to meet the promise of individualized treatment and health care.
The problem of identifying the optimal DTRs that maximize the long-term clinical outcome using reinforcement learning2
has received much attention in the statistics community3–11 . The existing DTR methods are proposed on Sequential Multiple
Assignment Randomized Trial (SMART)12 , in which the methods for DTR optimization are limited to clearly defined homogeneous decision stages and low-dimensional action spaces. They are difficult to implement using observational data (such
as electronic medical records, registry data), which exhibit much higher degree of heterogeneity in decision stages among
patients, and the treatment options (i.e., the action space) are often high-dimensional. The existing methods can only analyze
certain simplification of stage and action spaces among the enormous ways. Simplification by human experts might not lead
to the optimal DTRs and in many cases there is no clear way of simplification. In addition, the simplification process needs
substantial domain knowledge and labor-intensive data mining and feature engineering processes. There is a call for methods to
expand DTR methodology from the limited application in SMART studies to broader, flexible, and practical applications using
registry and cohort data.
To make reinforcement learning accessible for more general DTR problems using observational datasets, we need a new
framework which (i) automatically extracts and organizes the discriminative information from the data, and (ii) can explore
high-dimensional action and state spaces and make personalized treatment recommendations. Deep learning is a promising new
technique to use representation learning and save the labor-intensive feature engineering processes. The effective combination
of deep learning (deep neural networks) and reinforcement learning technique, named Deep Reinforcement Learning (DRL),
is initially invented for intelligent game playing and has later emerged as an effective method to solve complicated control
problems with large-scale, high-dimensional state and action spaces13–18 . The deep learning and DRL methods are promising
to automatically extract discriminate information among decision stages, patient features, and treatment options. In the work

we incorporate the state-of-the-art deep reinforcement learning into the DTR methodology and propose the first (to the best
of our knowledge) data-driven framework that is scalable and adaptable to optimizing DTR with high-dimensional treatment
options, and heterogeneous decision stages.
To demonstrate the effectiveness of the proposed framework, we implemented it using a concrete example: Graft Versus
Host Disease (GVHD) prevention and treatment for Leukemia patients who have undergone allogeneic hematopoietic cell
transplantation (AHCT). The long-term longitudinal follow-up for almost all US patients and some international patients who
have undergone AHCT make the Center for International Blood and Marrow Transplant Research (CIBMTR) registry database
an ideal existing data set to explore the capacity of artificial intelligence in medical decision making.
Reference19 points out that GVHD is a major complication of AHCT. Once established, GVHD is difficult to treat. It
can be prevented by selected methods, but often at the expense of an increased risk of relapse, rejection or delayed immune
reconstitution20, 21 . Hence, no optimal or even satisfactory prevention and treatment methods have been defined. Reference19
concluded that the difficulty in composing a standard practice guideline is the lack of solid scientific support for a large portion
of procedures used in GVHD prevention and treatment, which calls for further systematic studies to compare different strategies.
Such clinical needs for methodological innovations in finding the optimal personalized strategies can be largely resolved in the
proposed study.
More specifically, in this paper we develop a data-driven deep reinforcement learning (DRL) framework for the optimal DTR,
comprising the prevention and treatment of both acute and chronic GVHDs, as well as the initial conditioning (chemotherapy)
after the transplantation. The DRL framework, which deals with heterogenous decision stages (states) and high-dimensional
action space, consists of two steps at each decision stage. The first step is to build a deep neural network to predict experts’
treatment with a high-dimensional action space. The second step is to estimate the value function of DTRs for strategies
composed of the top expert actions with highest probability from the first step. The state and action spaces as well as reward
function are carefully selected, and effective dimensionality reduction techniques such as the low variance filter are utilized
to mitigate the shortcoming of limited data in the database. The similar states have similar encoded representations. In the
experimental results, we have demonstrated promising accuracy in predicting human experts’ decisions, as well as the high
expected reward function in the DRL-based dynamic treatment regimes.

Results
In this section, we present results on the deep neural networks’ prediction accuracy for expert treatment as well as the
performance of the deep reinforcement learning for optimizing the sequence of treatments. Experiments are conducted based
on the CIBMTR registry with data of 6,021 patients. The initial conditioning (to prevent relapse) and GVHD prophylaxis (to
prevent GVHD) were administered right before the transplant, thus they are considered as action at time t = 0; the treatment
of acute GVHD takes place at 100 days and 6 months (180 days); the treatment of chronic GVHD takes place at 6 months, 1
year (365 days), 2 years (730 days), and 4 years. We test DTR within 4 years after transplantation because a large portion of
patients’ data will be missing after that time, and live patients without relapse can be considered to be cured from the disease.
In the following, we will demonstrate the data-driven experimental results on the first step, i.e., building a deep neural network
to predict experts’ treatment, and then the second step, i.e., DRL-based framework of value function estimation and making
recommendations among treatment options. We adopt separate DNNs for predicting experts’ treatment in the first step, and
separate DRLs for treatment of acute and chronic GVHDs in the second step. This is because of the limited data size to train an
overall large DNN or DRL model. Details of the proposed procedure will be discussed in the next section.

Results on Predicting Experts’ Treatment
First, we demonstrate in Figure 1 the prediction accuracies of the initial conditioning and the initial GVHD prevention
(prophylaxis). We use 80% of the data set as training data and the remaining 20% for testing data, which is common for the
deep learning testings. Please note that we utilize the top-N prediction accuracy, i.e., the prediction is valid as long as the actual
treatment action from human experts is among the top N choices suggested by the deep neural network. This top-N accuracy is
widely utilized for the image recognition such as the ImageNet contest22 and other deep learning testings. We can observe that
(i) the top-N accuracy is in general between 75% and 90%, which shows the effectiveness of the proposed method; and (ii) the
top-N accuracy will increase with the increase of the N value.
2/11

(a) Initial conditioning

(b) Initial GVHD prophylaxis

Figure 1. Accuracies on predicting experts’ treatment for initial conditioning and GVHD prophylaxis.
Furthermore, Figure 2 illustrates the top-N prediction accuracy results for acute GVHD treatments at (a) 100 days and (b) 6
months. Figure 3 illustrates the (a) top-7 and (b) top-10 prediction accuracy results for chronic GVHD treatments, at 100 days,
6 months, 1 year, and 2 years. Again we use 80% of the data set as training data and the remaining 20% for testing data. From
these two figures we can derive the following observations. First, the prediction accuracies are in general higher compared
with the initial conditioning and GVHD preventions, because the medication for GVHD treatments seems to be more regular
compared with the initial treatments. The prediction accuracies are high enough and this shows a first step towards the ultimate
goal of DTR using machine intelligence. Next, for the chronic GVHD treatment, the prediction accuracy will increase when
time elapses, i.e., the prediction accuracy at 180 days is higher than that at 100 days, and the accuracy at 1 year will be even
higher. The reason is that the patients will become more stable and easy for treatment when chronic GVHD occurs or prolongs
at a later time.

(a) Results at 100 days

(b) Results at 6 months

Figure 2. Accuracies on predicting experts’ treatment for acute GVHD.

(a) Top-7 accuracy results

(b) Top-10 accuracy results

Figure 3. Accuracy results on predicting experts’ treatment for chronic GVHD at 100 days, 6 months, 1 year, and 2 years.
3/11

Results on DRL-based Value Function Estimation and Making Recommendations

In this section, we provide experimental results on the effectiveness of the DRL-based DTR framework for acute and chronic
GVHD treatments, i.e., using DRL for value function estimation and making recommendations. Because of the limited data
size to train an overall large DRL model, we build separate DRL models for the treatments of acute and chronic GVHDs.
Details about the DRL models are provided in the next section. Again we use 80% of the data set as training data and the
remaining 20% for testing data.

The reader of the paper will be most interested in how the DRL-based recommendation making would improve the
cumulative outcome, i.e., the disease free survival, of the patients. As a result, we compare between the proposed DRL-based
approach with random action selection baseline in terms of the value function. The details of value function are described in the
next section (we use the highest value 1 for relapse-free and GVHD-free survival, the lowest value 0 for death, and values in
between for other terminal states such as relapse, GVHDs, etc.) More specifically, the baseline uses the average DRL values of
all available actions (excluding the one with the highest expected reward) to mimic a random action selection policy. Using the
actual values from he observational data set results in similar baseline performance and will not be shown in this paper.

Figure 4 illustrates the comparison results between the proposed DRL method and baseline for acute GVHD treatment,
while Figure 5 shows the comparison results for chronic GVHD treatment. Despite the limited data, we can still observe that
the proposed DRL method outperforms the baseline methods both for acute and chronic GVHD treatments, which illustrates
the effectiveness of using the DRL method for making recommendations in DTR. Also, we can observe that the value function
(accumulative rewards) will increase when time elapses. This observation can also be explained as the expected outcome (e.g.,
the final relapse-free survival rate) will become higher for a patient if he/she has survived without relapse over a period of time
(say 1 year or 2 years).

Figure 4. Comparison results between the proposed DRL method and the baseline (details in the context) for acute GVHD
treatment.
4/11

Figure 5. Comparison results between the proposed DRL method and the baseline (details in the context) for chronic GVHD
treatment.

Discussion
In this work, we present a machine learning strategy from an observational dataset to address the decision making problem of
GVHD prevention and treatment. It is of significant interests to incorporate this machine learned rule to facilitate treatment
decision making and how to update the decision rules in an online fashion when new data are collected. There are some current
trends in the mobile health field that combines randomized clinical trial with online reinforcement learning through microrandomized-trials23 , where the randomization probability can be adopted in an online manner, in analogy to the exploration
techniques in reinforcement learning. The applications can be seen in smoking secession, eating disorder management, and
blood glucose management for diabetes patients. However, compared with our motivating example in bone marrow transplant,
these existing interventions are easier to perform randomization due to the much fewer number of actions, less profound
consequences, fewer treatment options and less complicated feature variables.
Nevertheless, in the clinical fields like our motivating example, there are some pressing sequential decision making questions.
For example, in the leukemia field, one other example is to decide whether transplant is a beneficial strategy compared to
non-transplant, under what condition or time transplant will become a better option, and adapting these decisions to personal
features. Given the constraints on conducting sequential randomized clinical trials on these questions, it is more practical
to start from analyzing the observational data at this point. With the improvement of data collection and machine learning
techniques in this field, a data-driven decision support system can provide treatment recommendations for doctors based on
supervised learning and reinforcement learning. Furthermore, one can adopt the exploration policy in reinforcement learning
for the adaptive treatment recommendations, while the decision is made through doctors and patient’s preference.
The Q-learning has theoretical guarantee to converge to optimal policy only under the assumptions for Markov decision
process. However Deep Q-learning does not have theoretical guarantee for convergence to optimal policy even under the
Markov decision process because of the sub-optimality of deep neural networks. The disease progression process does not
strictly follow Markov process and the four state variables we are considering may not fully capture the patients status. However
Q-learning and DQN have demonstrated good performance in a lot of application that Markov (memoriless) property does not
hold24, 25 . For future work we will remedy for this problem with model without Markov assumption (e.g. RNN), taking the
history information into account.

Methods
In this section we discuss in details the proposed DRL framework for the optimal DTR, comprising the prevention and treatment
of both acute and chronic GVHDs, as well as the initial conditioning after transplantation. We first provide a general framework
of DRL which can deal with complicated control problems with high-dimensional state spaces, and then the cohort retrieving
and data pre-processing, problem formulation, state and action spaces, reward function, and optimization techniques of the
proposed DRL framework for precision medicine.
The General DRL Framework for Complicated Control Problems
The general DRL framework, which can be utilized to solve complicated control problems, consists of two phases: an offline
deep neural network (DNN) construction phase and an online deep Q-learning phase13–15 . In the offline phase, a DNN is
5/11

utilized to derive the correlation between each state-action pair (s, a) of the system under control and the corresponding value
function Q(s, a). Q(s, a) represents the expected cumulative and discounted reward when the system starts from state s and
follows action a and certain policy thereafter. Q(s, a) for a discrete-time system is given by:
h ∞
i
Q(s, a) = E ∑ γ k r(k) s0 , a0

(1)

k=0

where r(t) is the reward rate and γ is the discount rate in a discrete-time system.
In order to construct a DNN with a good accuracy, the offline phase needs to accumulate enough samples of Q(s, a)
value estimates and the corresponding state-action pairs (s, a). It can be a model-based procedure or obtained from actual
measurement data15 , in which the latter is the case for optimal DTRs in precision medicine. This procedure includes simulating
the control process, and obtaining the state transition profile and estimations for Q(s, a) value, using an arbitrary but gradually
refined policy. The state transition profile is stored in an experience memory D with capacity ND . According to the inventors of
DRL14 , the use of experience memory can smooth out learning and avoid oscillations or divergence in the parameters. Based on
the stored state transition profile and Q(s, a) value estimates, the DNN is constructed with weight set θ trained using standard
training algorithms such as backpropagation based stochastic gradient descent algorithms. The overall procedure is shown in
the first part of Algorithm 1.
Algorithm 1 Illustration of the General DRL Framework
Ensure: This is the offline part
1: Extract real data profiles using certain control policies and obtain the corresponding state transition profiles and Q(s, a)
value estimates;
2: Store the state transition profiles and Q(s, a) value estimates in experience memory D with capacity ND ;
3: Iterations may be needed in the above procedure;
4: Offline: Pre-train a DNN with features (s, a) and outcome Q(s, a);
Require: This is the online part
5: for each execution sequence do
6:
for at each decision epoch tk do
7:
With probability ε select a random action, otherwise ak = argmaxa Q(sk , a), in which Q(sk , a) is derived (estimated)
from DNN;
8:
Perform system control using the chosen action;
Observe state transition at next decision epoch tk+1 with new state sk+1 , receive reward rk (sk , ak ) during time period
9:
[tk ,tk+1 );
10:
Store transition (sk , ak , rk , sk+1 ) in D;
11:
Updating Q(sk , ak ) based on rk (sk , ak ) and maxa0 Q(sk+1 , a0 ) based on Q-learning updating rule;
12:
end for
13:
Update DNN parameters θ using new Q-value estimates;
14: end for
For the online phase, the deep Q-learning technique is utilized based on the offline-trained DNN to select actions and update
Q-value estimates. More specifically, at each decision epoch tk of an execution sequence, suppose the system under control
is in the state sk . The DRL agent performs inference using DNN to obtain the Q(sk , a) value estimate for each state-action
pair (sk , a). Then according to the ε-greedy policy, the action with the maximum Q(sk , a) value estimate is selected with a
1 − ε probability and a random action is selected with an ε probability. After choosing an action denoted by ak , the DRL agent
receives total reward rk (sk , ak ) during [tk ,tk+1 ) before the next decision epoch tk+1 , and this leads to Q-value updates. The
reference work has proposed to utilize a duplicate DNN Q̂ for Q-value estimate updating, in order to mitigate the potential
oscillation of the inference results of the DNN26 . At the end of the execution sequence, the DNN is updated by the DRL agent
using the recently observed Q-value estimates in a mini-batch manner, and will be employed in the next execution sequence.
The overall procedure is shown in the second part of Algorithm 1.
As can be observed from the above procedure, the DRL framework is highly scalable for problems with a large state space,
which is distinctive from the traditional reinforcement learning techniques. On the other hand, the DRL framework requires an
enumerable action space due to the fact that at each decision epoch the DRL agent needs to enumerate all possible actions at
the current state and perform inference using the DNN to derive the optimal Q(s, a) value estimate (and corresponding optimal
action). This implies that the action space in the general DRL framework, or for the specific optimal DTR problem, needs to be
effectively reduced.
6/11

Table 1. Matching Information of Patients and Donors in the Data Set of Interests
Identical Sibling
3877

Other Relative
451

URD Well Matched
686

URD Partially Matched
433

URD Mismatched
173

Other
401

Developing a DRL Framework to Derive the Optimal DTR
In this section, we present the developed DRL framework with a motivating DTR application using the database from CIBMTR
registry on prevention and treatment of GVHD. There are two forms of GVHD: acute GVHD typically occurs within the first 6
months after the transplant and lasts for a short term if successfully treated; chronic GVHD may occur from shortly after the
transplantation to a few years later, and often requires long-term treatment that can lead to long-term complications/mobidity.
Throughout this paper, we denote the time index t = 0 for the time of transplantation, t = 1 for 100 days, t = 2 for 6 months,
t = 3 for 1 year, t = 4 for 2 years, and t = 5 for 4 years. We consider DTR within 4 years after the transplantation. In this paper,
we adopt the DRL technique for three tasks of DTR: initial treatment before the transplantation including initial conditioning
(chemotherapy to prevent relapse) and GVHD prophylaxis (to prevent GVHD), treatment of acute GVHD, and treatment of
chronic GVHD. The initial preventive treatments take place at the time of transplantation t = 0; the treatment of acute GVHD
takes place at times t = 1 (100 days) and t = 2 (6 months); the treatment of chronic GVHD takes place at times t = 2 (6 months)
through t = 5 (4 years).
As can be observed in Figure 7, the proposed DRL framework for optimal DTR comprises two steps at each decision
epoch/stage. The first step is to build a supervised learning network to predict the distribution of human experts’ decisions on
treatment actions. The second step is to estimate the value functions for treatment decisions with high probabilities (the actual
implementation is also compatible with estimating value functions for all treatment options). In this way the proposed framework
can both provide human experts’ opinions and the data-driven comparisons of different strategies and recommendation for the
optimal strategy, with relatively minor computational efforts. The proposed DRL framework is data-driven and scalable to
the heterogeneous decision stages and high-dimensionality in patient features and treatment options. The DRL framework
is adaptive, in that models in both steps will be updated when new data comes that correspond to new patients or treatment
outcomes.
Retrieving the Target Cohort and Pre-Processing Data

The cohort of patients used for this analysis consists of 6,021 patients diagnosed with Acute Myeloid Leukemia (AML) who
have undergone HCT between 1995 and 2007. Due to the discrete data collection scheme, we have higher quality data on
the onsets of GVHD conditions and the subsequent treatment decisions in a discrete-time frame indicating the occurrence
between two follow-up times. The exact date and sequence of treatment decisions between two periods of time are missing or
not recorded to a greater extent. In this work, the state and action are considered to be the state and action taken at the time each
form was recorded. We consider relapse and death as terminal states and occurrences of acute or chronic GVHD as transient
states. We consider baseline features of patients and donors that have been shown to affect GVHD and survival rates in clinical
studies, including patient’s age, gender, and co-morbidity information (including diabetes, seizure, hypertension, etc.). It also
includes important patient and donor relationship to patient and matching information, as well as donor’s gender. This cohort
includes both pediatric and adult patients. We include the histogram of age in Figure 6. And the Human Leukocyte Antigen
(HLA) matching results of patients are presented in Table 1.
Building a Deep Neural Network to Predict Expert Treatments
As shown in Figure 7, the first step at each decision epoch is to build a supervised learning network to predict the distribution of
human experts’ decisions on treatment actions. The prediction networks are illustrated in Figure 7. For the initial treatment
before the transplantation, the input features (the state space) include the union of the basic information of patients (e.g., age,
gender, and cormorbilities, etc.) and the HLA matching information between the patient and the donor. The output label
(action) is the combination of medicines to be utilized for the initial treatment which include the initial conditioning to avoid
disease relapse and the GVHD prophylaxis to prevent GVHD.
For the treatment of acute GVHD at time stamps t = 1 and t = 2, the input features (the state space) include both the basic
information of patients and the pairing conditions, as well as whether the patient has acute GVHD at that specific time stamp.
The output label (action) is the combination of medicines to be utilized for the treatment of acute GVHD. Similar input features
and actions also apply to the treatment of chronic GVHDs from t = 2 through t = 5.
To reduce the high dimensionality in the action space, we encode the actions using all the medicine combinations that have
been already utilized by doctors. We adopt an effective encoding scheme of the state space to reduce the state space to a large
extent, thereby accelerating the convergence speed and mitigating potential overfitting issues. For enhancing the accuracy,
separate multi-layer deep neural networks (instead of a single integrated network) are trained offline for the initial conditioning,
7/11

Figure 6. Histogram of Patient Ages in the Data Set of Interests

.
prevention of GVHDs, treatment of acute and chronic GVHDs.
In this step, we adopt the multiple layer, fully-connected neural network as our deep neural network. The network
architecture consists of four layers: the input layer, two hidden layers and the output layer. The dimension of the input layer is
9 and the two hidden layers have 16 and 32 neurons, respectively. The output dimension is 145 for initial conditioning and 127
for GVHD prophylaxis. The output dimensions for treating acute and chronic GVHDs are 283 and 271, respectively. We use
the Adam Optimizer to train the network, and the learning rate η is set to be 10−427 .
Estimating Value Function for Top Expert Choices and Making Recommendations

As shown in Figure 7, the second step is to estimate the value function for expert actions with highest probabilities and
make recommendations among treatment options. Our recommender only evaluates value function for actions with highest
probabilities, since actions with small probability have too small number of samples in the observational medical datasets
to arrive at a general conclusion. This restriction also reduces the computational complexity. The reward/outcome of major
interests is the relapse-free survival time after the transplantation, denoted as Ti . Let ~a denote the vector of actions at all stages,
~π denote the rules of decision sequences (i.e., policies), which represent the mapping from the currently observed state to
action at each stage. The value function of a policy ~π is V (~π ) = E(Ti |~a ∈ ~π (s)). The objective is to maximize V (~π ) and the
so-called Q-function is the expected reward if a subject (patient) is assigned to the optimal treatment in all the future stages,
and can be estimated through Dynamic Programming following the ideas from Q-learning28 . The learning algorithm will be
tailored for the specific Censoring Scheme of the data set. Denote Mi as the indicator of whether patient i is censored (Mi = 1 if
death or relapse is observed), and Ci as the last observation time of patient i. Denote Dt,i as the indicator that death or relapse is
observed within the time period t − 1 to t. For time t and patient i, denote the indicator of observed terminal events at time
t as Mt,i = I(D1,i = 0, . . . , Dt−1,i = 0, Dt,i = 1), where I(·) is the indicator function. The general Q-learning uses a backward
induction procedure across time stamps (decision stages). At stage t, each valid training sample (patient) i needs to satisfy
Ci > t, and requires that action at to be observed. For patients with Mt+1,i = 1, we use their observed Ti as the outcome. For
patients with Ci > t + 1, or Ci = t + 1, Mi = 0, we use the estimated Q-function for the future stage as the outcome. In other
words, we impute patients who have survived beyond time stamp t + 1 using their optimal future value estimation, regardless of
censoring.
Besides the value function of relapse-free survival time, we also propose to use an alternative discretized value function
as shown in the following. For each patient i, let ti denote the time when he/she enters the terminal state (death, relapse, or
relapse-free survival after 4 years) or when his/hers data get lost. The delayed reward (outcome) of patient i at time ti can be
classified into the following categories:
1. Relapse-free and GVHD-free survival.
8/11

Figure 7. The proposed DRL framework for prevention and treatment of GVHD, as well as initial conditioning.
2. Survival with acute or chronic GVHD.
3. Relapse of the leukemia disease.
4. Death.
5. Data loss.
We assign different delayed rewards/outcomes for the five cases. For relapse-free and GVHD-free survival in 4 years, the highest
reward (1) is achieved. Survival with acute or chronic GVHD receives a slightly degraded reward (0.8). Relapsed patients
receive a significantly degraded reward (0.2). Death receives zero. This reward can be viewed as a heuristic 4-year-survival
probability adjusted for the quality-of-life. The missing data problem caused by the lost of follow-up is solved by the imputation
method as discussed above.
In order to accommodate high dimensionality in state and action spaces, the recent DRL literature implemented Q-learning
with deep neural networks to approximate the Q-function, which is named as the Deep Q-Network. In this problem of interest,
three separate deep Q-networks are developed for DTRs of initial conditioning (chemotherapy and prevention of GVHDs),
treatment of acute and chronic GVHDs. For the inputs of deep Q-networks at time t, the corresponding input states described in
the previous section (predicting human experts’ decisions) serve as states, and the predicted human experts’ decisions serve as
actions. Effective encoding scheme is utilized to reduce the input state space. The output prediction is the expected value/return
starting at this state and taking the corresponding action. Multi-layer deep neural networks are constructed to achieve this goal,
and only those patients whose data are available at each time t are utilized to train the deep Q-networks.
In the deep Q-network, we use a replay buffer to store the dataset26 . The replay buffer is a finite sized cache which can
store the sampled transition tuples (st , at , rt , st+1 ), and it discards the oldest samples when it is full. The reply buffer allows
the algorithm to benefit from learning across a set of uncorrelated transitions. Direct implementation of deep Q-learning may
cause the network to be unstable during the training process. As a result, we adopt the target network introduced in reference26 .
The target network is a copy of the Q-value network and is used to perform inference of Q(sk+1 , a0 ). The weights of the target
9/11

network are updated by slowly tracking of the updated parameters in the Q-value network: θ 0 ← τθ + (1 − τ)θ 0 with τ  1.
This constraint can significantly improve the stability of learning.
Similar to the first step, a four layer fully-connected neural network architecture is adopted in the DRL network for acute
and chronic GVHD treatments. It consists of the input layer, two hidden layers and the output layer. The input dimensions for
treating both acute and chronic GVHDs are 8. The output dimensions for treating acute and chronic GVHDs are 283 and 271,
respectively. The numbers of neurons in the two hidden layers are 32 and 64 for both acute and chronic GVHD treatments. The
learning rate η is set to be 10−3 . The target network updating parameter τ is set as 0.01, and the discount rate of reward γ is set
as 0.99. The size of replay buffer is 20000.

References
1. Lavori, P. W. & Dawson, R. A design for testing clinical strategies: biased adaptive within-subject randomization. J. Royal
Stat. Soc. Ser. A (Statistics Soc. 163, 29–38 (2000).
2. Sutton, R. S. & Barto, A. G. Reinforcement learning: An introduction, vol. 1 (MIT press Cambridge, 1998).
3. Moodie, E. E., Richardson, T. S. & Stephens, D. A. Demystifying optimal dynamic treatment regimes. Biom. 63, 447–455
(2007).
4. Lavori, P. W. & Dawson, R. Dynamic treatment regimes: practical design considerations. Clin. trials 1, 9–20 (2004).
5. Murphy, S. A. Optimal dynamic treatment regimes. J. Royal Stat. Soc. Ser. B (Statistical Methodol. 65, 331–355 (2003).
6. Robins, J. M. Optimal structural nested models for optimal sequential decisions. In Proceedings of the Second Seattle
Symposium in Biostatistics, 189–326 (Springer, 2004).
7. Zhang, B., Tsiatis, A. A., Laber, E. B. & Davidian, M. A robust method for estimating optimal treatment regimes. Biom.
68, 1010–1018 (2012).
8. Zhao, Y., Kosorok, M. R. & Zeng, D. Reinforcement learning design for cancer clinical trials. Stat Med. 28, 3294–3315
(2009).
9. Murphy, S. A., Oslin, D. W., Rush, A. J. & Zhu, J. Methodological challenges in constructing effective treatment sequences
for chronic psychiatric disorders. Neuropsychopharmacol. 32, 257–262 (2006).
10. Zhao, Y., Zeng, D., Laber, E. & Kosorok, M. R. New statistical learning methods for estimating optimal dynamic treatment
regimes. J. Am. Stat. Assoc. DOI: 10.1080/01621459.2014.937488 (2014).
11. Liu, Y., Wang, Y., Kosorok, M. R., Zhao, Y. & Zeng, D. Robust hybrid learning for estimating personalized dynamic
treatment regimens. arXiv preprint arXiv:1611.02314 (2016).
12. Murphy, S. A. An experimental design for the development of adaptive treatment strategies. Stat. medicine 24, 1455–1481
(2005).
13. Mnih, V. et al. Playing atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602 (2013).
14. Mnih, V. et al. Human-level control through deep reinforcement learning. Nat. 518, 529–533 (2015).
15. Silver, D. et al. Mastering the game of go with deep neural networks and tree search. Nat. 529, 484–489 (2016).
16. Wei, T., Wang, Y. & Zhu, Q. Deep reinforcement learning for hvac control in smart buildings. In Design Automation
Conference (DAC) (ACM/IEEE, 2017).
17. Liu, N. et al. A hierarchical framework of cloud resource allocation and power management using deep reinforcement
learning. In International Conference on Distributed Computing Systems (IEEE, 2017).
18. Xu, Z., Wang, Y., Tang, J., Wang, J. & Gursoy, M. C. A deep reinforcement learning based framework for power-efficient
resource allocation in cloud rans. In International Conference on Communications (IEEE, 2017).
19. Ruutu, T. et al. Prophylaxis and treatment of gvhd: Ebmt–eln working group recommendations for a standardized practice.
Bone marrow transplantation 49, 168–173 (2014).
20. Bacigalupo, A. et al. Increased risk of leukemia relapse with high-dose cyclosporine a after allogeneic marrow transplantation for acute leukemia. Blood 77, 1423–1428 (1991).
21. Patterson, J. et al. Graft rejection following hla matched t-lymphocyte depleted bone marrow transplantation. Br. journal
haematology 63, 221–230 (1986).
22. Deng, J. et al. Imagenet: A large-scale hierarchical image database. In Computer Vision and Pattern Recognition, 2009.
CVPR 2009. IEEE Conference on, 248–255 (IEEE, 2009).
10/11

23. Klasnja, P. et al. Microrandomized trials: An experimental design for developing just-in-time adaptive interventions. Heal.
Psychol. 34, 1220 (2015).
24. Liu, N. et al. A hierarchical framework of cloud resource allocation and power management using deep reinforcement
learning. In Distributed Computing Systems (ICDCS), 2017 IEEE 37th International Conference on, 372–382 (IEEE,
2017).
25. Zhu, Y. et al. Target-driven visual navigation in indoor scenes using deep reinforcement learning. In Robotics and
Automation (ICRA), 2017 IEEE International Conference on, 3357–3364 (IEEE, 2017).
26. Lillicrap, T. P. et al. Continuous control with deep reinforcement learning. arXiv preprint arXiv:1509.02971 (2015).
27. Kingma, D. P. & Ba, J. Adam: A method for stochastic optimization. CoRR abs/1412.6980 (2014). URL http:
//arxiv.org/abs/1412.6980. 1412.6980.
28. Watkins, C. J. & Dayan, P. Q-learning. Mach. learning 8, 279–292 (1992).

Acknowledgement
Our thanks to the CIBMTR to provide the dataset. The CIBMTR is supported primarily by grants NCI/NHLBI/NIAID
5U24CA076518, and NCI/NHLBI 5U10HL069294.

Competing financial interests
The author(s) declare no competing financial interests.

Author contribution statement
Ying Liu initiated the idea after discussion with Yanzhi Wang. Brent Logan helped identified the clinical data set and give
guidance on the formulation of dynamic treatment regimes and data cleaning. Ying Liu cleaned the data and aligned the data
set in terms of state and actions for reinforcement learning framework. Ning Liu mainly designed the structure of the neural
networks and deep Q networks and conducted the analysis. Zhiyuan Xu and Jian Tang also helped with the data analysis. Ying
Liu, Yanzhi Wang and Ning Liu write different part of the manuscript.

Availability of materials and data
The authors are willing to share the cleaned data and code with the Editorial Board Members and referees upon request.
The cleaned data are identified human subject information summarized in forms of state and actions at each decision point,
additional steps of IRB approval maybe needed in accordance with CIBMTR registry policy.

11/11

