Robust Actor-Critic Contextual Bandit for Mobile Health (mHealth) Interventions
Feiyun Zhu Jun Guo Ruoyu Li Junzhou Huang

𝐀
 𝐚
 
 

𝐀
 𝐚
 𝒜
 𝔸
 

𝐁
 𝐛
 
 
 

𝐁
 𝐛
 ℬ
 𝔹
 

𝐂
 𝐜
 
 

𝐂
 𝐜
 𝒞
 ℂ
 

𝐃
 𝐝
 
 
 

𝐃
 𝐝
 𝒟
 𝔻
 

𝐄
 𝐞
 
 

𝐄
 𝐞
 ℰ
 𝔼
 

𝐅
 𝐟
 𝐅̅
 𝐟̅
 

𝐅
 𝐟
 ℱ
 𝔽
 

𝐆
 𝐠
 𝐆̅
 𝐠̅

𝐆
 𝐠
 𝒢
 𝔾
 

𝐇
 𝐡
 𝐇̅
 𝐡̅

𝐇
 𝐡
 ℋ
 ℍ
 

𝐈
 𝐢
 𝐈̅
 𝐢̅

𝐈
 𝐢
 ℐ
 𝕀
 

𝐉
 𝐣
 𝐉̅
 𝐣̅

𝐉
 𝐣
 𝒥
 𝕁
 

𝐊
 𝐤
 𝐊̅
 𝐤̅

𝐊
 𝐤
 𝒦
 𝕂
 

𝐋
 𝐥
 𝐋̅
 𝐥̅
 

𝐊
 𝐤
 ℒ
 𝕃
 

𝐌
 𝐦
 𝐌̅
 𝐦̅

𝐌
 𝐦
 ℳ
 𝕄
 

𝐍
 𝐧
 𝐍̅
 𝐧̅
 

𝐍
 𝐧
 𝒩
 ℕ
 

𝐎
 𝐨
 𝐎̅
 𝐨̅

𝐎
 𝐨
 𝒪
 𝕆
 

𝐏
 𝐩
 𝐏̅
 𝐩̅

𝐏
 𝐩
 𝒫
 ℙ
 

𝐐
 𝐪
 𝐐̅
 𝐪̅

𝐐
 𝐪
𝒬
 ℚ
 

𝐑
 𝐫
 𝐑̅
 𝐫̅

𝐑
 𝐫
ℛ
 ℝ

𝐒
 𝐬
 𝐒̅
 𝐬̅
 

𝐒
 𝐬
𝒮
 𝕊
 

𝐓
 𝐭
 𝐓̅
 𝐭̅

𝐓
 𝐭
 𝒯
 𝕋
 

𝐔
 𝐮
 𝐔̅
 𝐮̅

𝐔
 𝐮
 𝒰
 𝕌
 

𝐕
 𝐯
 𝐕̅
 𝐯̅

𝐕
 𝐯
𝒱
 𝕍
 

𝐖
 𝐰
 𝐖̅
 𝐰̅

𝐖
 𝐰
 𝒲
 𝕎
 

𝐗
 𝐱
 
 

𝐗
 𝐱
 𝒳
 𝕏
 

𝐘
 𝐲
𝐘̅
 𝐲̅
 

𝐘
 𝐲
 𝒴
 𝕐
 

𝐙
 𝐳
 𝐙̅
 𝐳̅
 

𝐙
 𝐳
𝒵
 ℤ
 

th

0
 1

Tr

TV

𝔼
 

det

𝐯𝐞𝐜
 

𝐯𝐚𝐫
 

𝐜𝐨𝐯
 

𝐬.𝐭.
 

for

rank
 

rankn
 

𝐝𝐢𝐚𝐠
 

𝐬𝐢𝐠𝐧
 

𝐥𝐨𝐬𝐬
 

when
 

where
 

if
 




Robust Actor-Critic Contextual Bandit for Mobile Health (mHealth)
Interventions
    Feiyun Zhu^†, Jun Guo^, Ruoyu Li^†,
Junzhou Huang^†† Department of CSE, University of Texas at Arlington, TX,
76013, USA.  Department of Statistics, Univeristy of Michigan,
Ann Arbor, MI 48109, USA
    December 30, 2023
=========================================================================================================================================================================================================


We consider the actor-critic contextual bandit for the mobile health
(mHealth) intervention. State-of-the-art decision-making algorithms
generally ignore the outliers in the dataset. In this paper, we propose
a novel robust contextual bandit method for the mHealth. It can achieve
the conflicting goal of reducing the influence of outliers, while
seeking for a similar solution compared with the state-of-the-art
contextual bandit methods on the datasets without outliers. Such performance
relies on two technologies: (1) the capped-ℓ_2 norm; (2) a
reliable method to set the thresholding hyper-parameter, which is
inspired by one of the most fundamental techniques in the statistics.
Although the model is non-convex and non-differentiable, we propose
an effective reweighted algorithm and provide solid theoretical analyses.
We prove that the proposed algorithm can find sufficiently decreasing
points after each iteration and finally converges after a finite number
of iterations. Extensive experiment results on two datasets demonstrate
that our method can achieve almost identical results compared with
state-of-the-art contextual bandit methods on the dataset without
outliers, and significantly outperform those state-of-the-art methods
on the badly noised dataset with outliers in a variety of parameter
settings.




§ INTRODUCTION


Due to the explosive growth of smart device (i.e. smartphones and
wearable devices, such as Fitbit etc.) users globally, mobile health
(mHealth) technologies draw increasing interests from the scientist
community <cit.>.
The goal of mHealth is to deliver in-time interventions to device
users, guiding them to lead healthier lives, such as reducing the
alcohol abuse <cit.>
and increasing physical activities <cit.>.
With the advanced smart technologies, the mHealth interventions can
be formed according to the users' ongoing statuses and changing needs,
which is more portable and flexible compared with the traditional
treatments. Therefore, mHealth technologies are widely used in lots
of health-related applications, such as eating disorders, alcohol
abuses, mental illness, obesity management and HIV medication adherence <cit.>.

Formally, the tailoring of mHealth intervention is modeled as a sequential
decision-making (SDM) problem. The contextual bandit algorithm provides
a framework for the SDM <cit.>.
In 2014, Lei <cit.> proposed
the first contextual bandit algorithm for the mHealth study. It is
in the actor-critic setting <cit.>,
which has two advantages compared with the critic only contextual
bandit methods for the internet advertising <cit.>:
(a) Lei's method has an explicit parameterized model for the stochastic
policy. By analyzing the estimated parameters in the learned policy,
we could know the key features that contribute most to the policy.
This is important to the behavior scientists for the state (feature)
design. (b) From the perspective of optimization, the actor-critic
algorithm has great properties of quick convergence with low variance <cit.>.

However, Lei's method assumes that the states at different decision
points are i.i.d. and the current action only influences the immediate
reward <cit.>. This assumption
is infeasible in real situations. Taking the delayed effect in the
SDM or mHealth for example, the current action influences not only
the immediate reward but also the next state and through that, all
the subsequent rewards <cit.>.
Accordingly, Lei proposed a new method <cit.>
by emphasizing on explorations and reducing exploitations.

Although those two methods serve a good start for the mHealth study,
they assume that the noise in the trajectory follows the Gaussian
distribution. The least square based algorithm is employed to estimate
the expected reward. In reality, however, there are various kinds
of complex noises that can badly degrade the collected data, for example:
(1) the wearable device is unreliable to accurately record the states
and rewards from users under different conditions. (2) The mobile
network is unavailable in some areas. Such case hinders the collecting
of users' states as well as the sending of interventions. (3) The
mHealth relies on the self-reported information (Ecological Momentary
Assessments, i.e. EMAs) <cit.> to deliver effective
interventions. However, some users are annoyed at the EMAs. They either
fill out the EMAs via random selections or just leave some or all
the EMAs blank. We consider the various kinds of badly noised observations
in the trajectory as outliers. 

There are several robust methods for the SDM problem <cit.>.
However, those methods are neither in the actor-critic setting, nor
focusing on the outlier problem. Thus, they are different from this
paper's focus. In the general machine learning task, there are some
robust learning methods to deal with the outlier problem <cit.>.
However, none of them are contextual bandit algorithms—it
may cost of lots of work to transfer their methods to the (actor-critic)
contextual bandit algorithms. Besides, those methods seldom pay attention
to the dataset without outliers. In practice, however, we don't know
whether a given dataset consists of outliers or not. It is necessary
to propose a robust learning method that works well on the dataset
both with and without outliers. 

To alleviate the above problems, we propose a robust contextual bandit
method for the mHealth.  The capped-ℓ_2 norm is used to measure
the learning error for the expected reward estimation (i.e. the critic
updating). It prevents outlier observations from dominating our objective.
Besides, the learned weights in the critic updating are considered
in the actor updating. As a result, the robustness against outliers
are greatly boosted in both actor-critic updatings. There is an important
thresholding parameter ϵ in our method. We propose a solid
method to set its value according to the distribution of samples,
which is based on one of the most fundamental ideas in the statistics.
It has two benefits: (1) the setting of ϵ becomes very easy
and reliable; (2) we may achieve the conflicting goal of reducing
the influence of outliers when the dataset indeed contains outliers,
while achieving almost identical results compared with the state-of-the-art
contextual bandit method on the datasets without outliers. Although
the objective is non-convex and non-differentiable, we derive an effective
algorithm. As a theoretical contribution, we prove that our algorithm
could find sufficiently decreasing point after each iteration and
finally converges after a finite number of iterations. Extensive experiment
results on two datasets verify that our methods could achieve clear
gains over the state-of-the-art contextual bandit methods for the
mHealth.



§  PRELIMINARIES


Multi-armed bandit (MAB) is the simplest algorithm for the sequential
decision making problem (SDM). The contextual bandit is a more practical
extension of the MAB by considering some extra information that is
helpful for the SDM problem <cit.>.
The use of context information allows for many interesting applications,
such as internet advertising and health-care tasks <cit.>. 

In contextual bandit, the expected reward is an core concept that
measures how many rewards we may averagely get when it is in state
s and choosing action a, i.e., (r| s,a).
Since the state space is usually very large or even infinite in the
mHealth tasks, the parameterized model is employed to approximate
the expected reward: (r| s,a)=(s,a)^⊺,
where (s,a)∈^u is a feature processing
step that combines information in the state s and the action a,
∈^u is the unknown coefficient vector. 

In 2014, Lei <cit.> proposed
the first contextual bandit method for the mHealth study. It is in
the actor-critic learning setting. The actor updating is the overall
optimization goal. It aims to learn an optimal policy π^* that
maximizes the average rewards over all the states and actions <cit.>.
The objective function is π_θ_n^*=max_θ_nJ(θ_n)
for the n^ user, where 

    J(θ_n)=∑_s∈d_ref^(n)(s)∑_a∈π_θ_n(a| s)(r| s,a)

and d_ref^(n)(s) is a reference
distribution over states for user n.

Obviously, we need the estimation of expected rewards (r| s,a)
to define the objective (<ref>) for the
actor updating. Such procedure is called the critic updating. State-of-the-art
method generally employs the ridge regression to learn the expected
reward from the observations. The objective is 

    min__n∑__i∈_n‖(s_i,a_i)^⊺_n-r_i‖ _2^2+ζ_c‖_n‖ _2^2,

where _n={_i=(s_i,a_i,r_i)| i=0,⋯,M}
is the trajectory of observed tuples from the n^ user and
_i is the i^ tuple in _n; ζ_c
is a tuning parameter to control the strength of constraints. It has
a closed-form solution for (<ref>)
as:

    _n=(_n_n^⊺+ζ_c_u)^-1_n_n.

where _u is a u× u identity matrix. Unfortunately,
similar to the existing least square based models in machine learning
and statistics, the objective function in (<ref>)
is prone to the presence of outliers <cit.>. 



§ ROBUST ACTOR-CRITIC CONTEXTUAL BANDIT VIA THE CAPPED-ℓ_2
NORM


To enhance the robustness in the critic updating, the capped-ℓ_2
norm based measure is used for the estimation of expected rewards.
By imposing the learned weights for the actor updating, we propose
a robust objective for the actor updating.



 §.§ Robust Critic Updating via the Capped-ℓ_2 Norm


To simplify the notation, we get rid of the subscript index n,
which is used to indicate the model for the n^ user. The
new objective for the critic updating (i.e. policy evaluation) is

    min_∑_i=1^Mmin{‖ r_i-_i^T‖ _2^2,ϵ} +ζ_c‖‖ _2^2,

where min{‖‖ _2^2,ϵ}
is the capped-ℓ_2 norm for a vector ; ϵ>0
is the thresholding hyper-parameter to choose the effective observations
for the critic updating; _i=(s_i,a_i)
is the feature for the estimation of expected rewards.

If the residual of the i^ tuple is ‖ r_i-_i^⊺‖ _2^2>ϵ,
 we treat it as an outlier. Its residual is capped to a fixed value
ϵ. That is, the influence of the i^ tuple is fixed <cit.>,
which can't cause bad influences on the learning procedure. For the
tuples whose residuals satisfy ‖ r_i-_i^⊺‖ _2^2≤ϵ,
we consider them as effective observations and keep them as they are
in the optimization process. 

Therefore, it is extremely important to properly set the value of
ϵ. When ϵ is too large, the outliers that distribute
far away from the majority of tuples will be treated as effective
samples, causing bad influences to the learning procedure. When ϵ
is too small, most tuples are treated as outliers—there
would be very few of effective samples for the cirtic learning. Such
case easily leads to some unstable policies that contain lots of variances.
Specially if ϵ→+∞, our objective is equivalent
to the least square objective in (<ref>). 

As a profound contribution, we propose a reliable method to properly
set the value of ϵ. Our method doesn't need any specific
assumption on the data distribution. It is derivated from the boxplot—one
of the most fundamental ideas in the statistics <cit.>.
To give a descriptive illustration of the data distribution, the boxplot
is widely used by specifying 5 points, including the min, lower quartile
q_1, median, upper quartile q_3 and max. Based on the 5
points, the boxplot provides a method to detect outliers. Following
this idea, we set the value of ϵ as 

    ϵ=τ(q_3+1.5× IQR)

where IQR=q_3-q_1 is the interquartile range; τ is introduced
only for the experiment setting S2, otherwise we may ignore
the parameter τ by keeping it fixed at 1. Intuitively, the
data points that are 1.5× IQR more above the third quartile
are detected as the outliers. Compared with the state-of-the-art robust
learning methods <cit.>
in the other fields that have to manually set the thresholding hyper-parameter,
we provide an adaptive method to set ϵ, which is well adapted
to the data distribution.

With the capped-ℓ_2 norm and the method to set ϵ
in (<ref>), we can  achieve the conflicting
goals of (a) reducing the influence of outliers when the dataset has
outliers, while (b) seeking for a similar solution compared with that
of the state-of-the-art method if there is no outlier in the dataset.
As a result, our method can deal with various datasets, regardless
of whether they consist of outliers or not. 



 §.§ Derivation of a General Objective
Function for (<ref>)


To give an efficient algorithm, we consider a more general capped-ℓ_2
norm based objective for (<ref>) as
follows

    min_∑_i=1^Mmin{‖ h_i()‖ _2^2,ϵ} +ζ g(),

where ‖·‖ _2^2 is the ℓ_2 norm
for a vector; h_i() and g()
are both scalar functions of . In this section, we propose
an iteratively re-weighted method to simplify the objective (<ref>).

Due to the non-smooth and non-differentiable property of (<ref>),
we could only obtain the sub-gradient of (<ref>)
as:

    ∂()=∑_i=1^M∂min(‖ h_i()‖ _2^2,ϵ)+ζ∂ g().

 Letting f=∂min(‖ h_i()‖ _2^2,ϵ)
gives 

    f=
    0     ‖ h_i()‖ _2^2>ϵ
    
    2[d_low,d_high]∂ h_i()     ‖ h_i()‖ _2^2=ϵ
    
    2h_i()∂ h_i()     ‖ h_i()‖ _2^2<ϵ,

where d_low=min{ 0,h_i()}
and d_high=max{ 0,h_i()}.
For the sake of easy optimization, we provide a compact expression
that satisfies the sub-gradient in (<ref>)
by introducing a variable u_i=1_{‖ h_i()‖ _2^2<ϵ}.
Then Eq. (<ref>) is rewritten as 

    ∂()=2∑_iu_ih_i()∂ h_i()+ζ∂ g().

Since u_i depends on , it is very challenging to directly
solve the objective (<ref>). Once
u_i is given for every i, the objective (<ref>)
is equivalent to the following problem 

    min_∑_i=1^Mu_i‖ h_i()‖ _2^2+ζ g()

in the sense that they have the same partial derivative.



 §.§ Robust Algorithm for the Critic Updating


In this section, we provide an effective updating rule for the objective
function (<ref>) (cf. Proposition <ref>
and Algorithm <ref>).
We prove that our algorithm can find sufficiently decreasing point
after each iteration (cf. Lemma <ref>)
and finally converge after a finite number of iterations (cf. Theorem <ref>).

The iterative updating
rule (<ref>) (∀ t>1) is 

    ^(t)=(^(t-1)^⊺+ζ_c)^-1^(t-1),

where ^(t)=(^(t))∈^M× M
is a nonnegative diagonal matrix. The i^ element is u_i^(t)=1_{‖ r_i-_i^⊺^(t)‖ _2^2<ϵ}.


The updating rule in Proposition <ref>
leads to sufficient decrease of the objective function ()
in (<ref>):

    f(^(t-1),^(t-1))≥ f(^(t),^(t))+ζ‖^(t)-^(t-1)‖ _2^2,

where the bivariate function f(,) is defined
as 

    f(,)=∑_i^Mu_i‖ r_i-_i^T‖ _2^2+∑_i^M(1-u_i)ϵ+ζ_c‖‖ _2^2,

which is the same as () in (<ref>).

  For {^(t),t≥0}
in Lemma <ref>, we show that ∑_t=1^∞‖^(t)-^(t-1)‖ _2^2<∞
and consequently 

    lim_t→∞‖^(t)-^(t-1)‖ _2^2=0.





The updating rule in Proposition <ref>
converges after a finite number of iterations.




 §.§ Robust Actor Updating for the Stochastic Policy


Besides the critic updating, outliers can also badly influence the
actor updating in (<ref>), which is our
ultimate objective. To boost its robustness, the estimated weights
learned in the critic updating are considered. Since d_ref(s)
is usually unavailable in reality, the M-trial based objective <cit.>
is widely used. Thus, the objective (<ref>)
is rewritten as 

    J(θ)=u_i/M∑_i=1^M∑_a∈π_θ(a| s_i)(r| s_i,a)-ζ_a/2‖θ‖ _2^2,

where (r| s_i,a)=(s_i,a)^⊺
is the estimated expected reward; ‖θ‖ _2^2
is the least square constraint to make the objective (<ref>)
a well-posed problem and ζ_a is a balancing parameter that
controls the penalization strength <cit.>.

Compared with the current objective for the actor updating in (<ref>),
our objective has an extra weight term { u_i} _i=1^M,
which gives those tuples, whose residuals are very large in the critic
updating, zero weights. As a result, the outlier tuples that are far
away from the majority of tuples are removed from the actor updating,
enhancing the robustness. The algorithm of the actor updating performs
the maximization of (<ref>) over θ.
This is learned via the Sequential Quadratic Programming (SQP) algorithm.
We utilize the implementation of SQP with finite-difference approximation
to the gradient in the fmincon function of Matlab. 

 



§ EXPERIMENT




 §.§ Two Datasets


Two datasets are used to verify the performance of our method. The
first dataset is on the personalizing treatment delivery in mobile
health, which is a common application of the sequential decision making
algorithm <cit.>. In
this paper, we focus on the Heartsteps, where the participants are
periodically sent activity suggestions aimed at decreasing sedentary
behavior <cit.>. Specifically, Heartsteps
is a 42-days trial study where 50 participants are involved. For each
participant, there are 210 decision points—five decisions
per participant per day. At each time point, the set of intervention
actions can be the intervention type, as well as whether or not to
send interventions. The intervention actions generally depend on the
state of the participant as well as the formerly sent interventions.
Interventions can be sent via smartphones, or via other wearable devices
like a wristband <cit.>.


The Heartsteps is a common application for the contextual bandit algorithm <cit.>.
The goal is to learn an optimal SDM algorithms to decide what type
of intervention actions to send to each user to maximize the cumulative
steps each user takes. The resulting data for each participant is
a sequence ={ s_0,a_0,r_0,⋯,s_209,a_209,r_209,r_210,s_210},
where s_t is the participant's state at time t. It is a three
dimensional vector that consists of (1) the weather condition, (2)
the engagement of participants, (3) the treatment fatigue of participants.
a_t indicates whether or not to send the intervention to users;
since the goal of Heartsteps is to increase the participant’s
activities, we define the reward, r_t, as the step count for
the 3 hours following a decision point.

The second dataset is the 4-state chain walk, which is a benchmark
dataset for the (contextual) bandit and reinforcement learning study.
Please refer to <cit.> for the details
of the chain walk dataset. The reward vector over the four states
is (0,100,100,0) in this paper. 




 §.§ Nine Compared Methods


There are nine contextual bandit methods compared in the experiment,
including three state-of-the-art methods: (1) Linear Upper Confidence
Bound Algorithm (LinUCB) is one of the most famous contextual bandit
methods used in the personalized news recommendation <cit.>;
(2) the actor-critic contextual bandit (ACCB) is the first SDM method
for the mHealth intervention <cit.>;
(3) the stochasticity constrained actor-critic contextual bandit (SACCB)
for the mHealth <cit.>.
We improve the above three methods by first using the state-of-the-art
outlier filter <cit.>
to remove outliers, then employing the above three contextual bandit
algorithms for the SDM task. In the 7^ to 9^ compared
methods, we apply the proposed robust model and optimization algorithms
in the three state-of-the-art contextual bandit methods, leading to
(7) Robust LinUCB (Ro-LinUCB for short), (8) Robust ACCB (Ro-ACCB),
(9) Robust SACCB (Ro-SACCB). 



 §.§ Evaluation Methodology and Parameter Setting


It has been a challenging problem to reliably evaluate the sequential
decision making (e.g., bandit and reinforcement learning) algorithms
when the simulator (like the Atari games) is unavailable <cit.>.
After the model is trained, we use it to interact with the enviroment
(or simulator) to collect thousands of immediate rewards for the calculation
of the long term rewards as the evaluation metric. However for a wide
variety of applications including our HeartSteps, the simulator is
unavailable. In this paper, we use a benchmark evalutaion methods <cit.>.
The main idea is to make use of the collected dataset to build a simulator,
based on which we train and evaluate the contextual bandit methods.
Such case makes it impossible to train and evaluate the contextual
bandit algorithms on up to ten datasets like the general supervised
learning tasks. 


In the Heartsteps study, there are 50 users; the simulator for each
user is as follows: the initial state is drawn from the Gaussian distribution
S_0∼_p{ 0,Σ}, where
Σ is a p× p covariance matrix with pre-defined elements.
For t≥0, a_t is drawn from the learned policy π_θ_n(a_t| s_t)
during the evaluation procedure. When t≥1, the state and immediate
reward are generated as 

    S_t,1=    β_1S_t-1,1+ξ_t,1,
    
    S_t,2=    β_2S_t-1,2+β_3A_t-1+ξ_t,2,
    
    S_t,3=    β_4S_t-1,3+β_5S_t-1,3A_t-1+β_6A_t-1+ξ_t,3,
    
    R_t=    β_13×[β_7+A_t×(β_8+β_9S_t,1+β_10S_t,2)
        +β_11S_t,1-β_12S_t,3+ϱ_t],

where β={β_1,⋯,β_13}=[0.4,0.3,0.4,0.7,0.05,0.6,3,0.25,0.25
,0.4,0.1,0.5,500] is the main parameter for the MDP system. {ξ_t,i} _i=1^3∼(0,1)
and ϱ_t∼(0,9) are the gaussian
noises in the state (<ref>) and
in the reward (<ref>) respectively.
 

The parameterized policy is assumed to follow the Boltzmann distribution
π_θ(a| s)=exp[-θ^⊺ϕ(s,a)]/∑_a'exp[-θ^⊺ϕ(s,a')],
where ϕ(s,a)=[as^⊺,a]^⊺∈^4
is the policy feature, θ∈^4 is the unknown coefficient
vector. The feature for the estimation of expected rewards is set
(s,a)=[1,s^⊺,a,s^⊺a]^⊺∈^8.
The ℓ_2 constraint for the actor-critic learning is set as
ζ_a=ζ_c=0.001. The outlier ratio and strength are set
ψ=4% and ν=5 respectively. In our methods, τ
is set as 1 by default. 

The expected long-run average reward (ElrAR) <cit.>
is used to quantify the quality of the estimated policy π_θ_n
for n∈{ 1,⋯,50}. Intuitively, ElrAR measures
the average steps users take per day in the long-run Heartsteps study
when we use the estimated policy π_θ_n to
send interventions to users. There are two steps to obtain the ElrAR:
(a) get the average reward η^π_θ_n for
the n^ user by averaging the rewards over the last 4,000
decision points in a trajectory of 5,000 tuples under the policy
π_θ_n; (b) the ElrAR 𝔼[η^π_θ̂]
is achieved by averaging over the 50 η^π_θ_ns. 



 §.§ Comparisons in the Four Experiment Settings


We carry out the following experiments to verify four aspects of the
contextual bandit methods: 

(S1) To verify the significance of the proposed method, we
compare nine contextual bandit methods on two datases: (1) HeartSteps
and (2) chain walk. The experiment results are summarized in Table <ref>,
where there are three sub-tables; each sub-table displays three methods
in a type: (a) the state-of-the-art contextual bandit, like ACCB;
(b) “OutlierFilter + ACCB” means that we first use the state-of-the-art
outlier filter <cit.>
to get rid of the outliers, then employ the state-of-the-art contextual
bandit method ACCB for the SDM task; (c) is the proposed robust contextual
bandit method (Ro-ACCB). As we shall see, “OutlierFilter” is helpful
to improve the performance of the state-of-the-art contextual bandit
methods. However, our three methods (i.e., Ro-LinUCB, Ro-ACCB and
Ro-SACCB) always obtain the best results compared with all the other
state-of-the-art methods in their type on both datasets. Compared
with the best state-of-the-art method, our three methods improve 131.4,
136.2 and 139.9 steps respectively on the HeartSteps. Although there
are lots of general outlier detection or outlier filter methods that
can be helpful to relieve the bad influence of outliers, it is still
meaningful to specifically propose a robust contextual bandit algorithm. 

(S2) In this part, the ratio of tuples ψ that contains
outliers rises from 0% to 9%. The experiment results are summarized
in Table <ref> and Figs. <ref>b, <ref>e.
In Table <ref>, there are two sub-tables, displaying
the ElrARs on the HeartSteps in the top, and the ElrARs on the chain
walk in the bottom. As we can see, when μ=0%, there is no outlier
in the trajectory. In such case, our results are (almost) identical
to that of LinUCB, ACCB and SACCB on both datasets. When ψ rises,
our results keep stable, while the ElrARs, of LinUCB, ACCB and SACCB,
decrease dramatically. Compared with ACCB, Ro-ACCB averagely achieves
an improvement of 10.5% on the HeartSteps dataset and 23.2%
on the chain walk dataset. Such results demonstrate that our method
is able to deal with the badly noised dataset that consists of a large
percentage (up to 9%) of outliers.

(S3) In this part, the strength of outliers ν ranges
from 0 to 10 times of the average value in the trajectory. The
experiment results are summarized in Table <ref>,
and Figs. <ref>c, <ref>f.
We have three observations based on the experiment results: (1) when
ν=0, there is no outlier in the trajectory. Out methods achieve
similar results with that of LinUCB, ACCB and SACCB; (2) when ν
rises in the domain, our method keeps stable on the HeartSteps and
decreases slightly on the chain walk. However, the results of LinUCB,
ACCB and SACCB decrease obviously when ν rises. Such phenomena
verify that our method is able to deal with the dataset both with
or without outliers. Besides, we may use our method on the dataset
with various strengths of outliers. 

(S4) In this part, the value of τ ranges from 0.1
to 8 on the HeartSteps dataset and from 0.2 to 4 on the chain
walk dataset. The experiment results are displayed in Figs. <ref>a
and <ref>d. As we shall see, the proposed method
obtains clear advantage over the state-of-the-art method, i.e., ACCB <cit.>,
in a wide range of τ settings. In average, our method improves
the ElrAR by 12.6% on the HeartSteps and 33.1% on the chain
walk, compared with the ACCB. Such results verify that the proposed
method to set τ is very promising. It is able to adapt to the
data property and select the effective, neither too few nor too many,
tuples in the trajectory for the actor-critic updating. Note that
ACCB does not have the parameter τ. Thus the result of ACCB
remains unchanged as τ rises.



§ CONCLUSION AND DISCUSSION


To deal with the outlier in the trajectory, we propose a robust actor-critic
contextual bandit for the mHealth intervention. The capped-ℓ_2
norm is employed to boost the robustness for the critic updating.
With the learned weights in the critic updating, we propose a new
objective for the actor updating, enhancing its robustness. Besides,
we propose a solid method to set an important thresholding parameter
in the capped-ℓ_2 norm. With it, we can achieve the conflicting
goal of boosting the robustness of our algorithm on the dataset with
outliers, and achieving almost identical results compared with the
state-of-the-art method on the datasets without outliers. Besides,
we provide theoretical guarantee for our algorithm. It shows that
our algorithm could find sufficiently decreasing point after each
iteration and finally converges after a finite number of iterations.
Extensive experiment results show that in a variety of parameter settings
our method achieves significant improvements. 




 §.§ Appendix 1: the proof of Proposition <ref> 


According to the analyses in Eqs (<ref>) and (<ref>),
we simplify (<ref>) into the following
objective 


    min_ ∑_i=1^Mu_i‖ r_i-_i^⊺‖ _2^2+ζ_c‖‖ _2^2.

Taking the partial derivative and setting it to zero give us the
updating rule 
 

    ^(t)=(^(t-1)^⊺+ζ)^-1^(t-1),

where ^(t)=(^(t))∈^M× M
is nonnegative diagonal. The i^ element is u_i^(t)=1_{‖ r_i-_i^⊺^(t)‖ _2^2<ϵ}.




 §.§ Appendix 2: the proof of Lemma <ref>


For t≥1, when fix ^(t-1), we find that

    f(,^(t-1))=    ∑_i^Mu_i^(t-1)‖ r_i-_i^T‖ _2^2+
       ∑_i^M(1-u_i^(t-1))ϵ+ζ_c‖‖ _2^2

is a quadratic function, which is strongly convex. The updating rule
in Proposition <ref> minimizes
f(,^(t-1)) globally over .
Via the strong convexity of f(,^(t-1)),
we have

    f(^(t-1),^(t-1))
    ≥    f(^(t),^(t-1))+ζ_c‖^(t)-^(t-1)‖ _2^2.

 When fixing ^(t), updating ^(t)
gives 

    f(^(t),^(t))-f(^(t),^(t-1))
    
    =     -∑_i^M|‖ r_i-_i^⊺‖ _2^2-ϵ|1_{ u_i^(t)≠ u_i^(t-1)}≤0.

Finally ∀ t≥1, we conclude the following inequation 

    f(^(t-1),^(t-1))
    ≥    f(^(t),^(t))+ζ_c‖^(t)-^(t-1)‖ _2^2.





 §.§ Appendix 3: the proof of Lemma <ref>


We sum up the function descent inequality (<ref>)
for t=1,2,⋯,T: 

    ∑_t=1^T‖^(t)-^(t-1)‖ _2^2
    ≤   2/ζ_c∑_i=1^T[O(^(t-1))-O(^(t))]
    
    =    2/ζ_c[O(^(0))-O(^(T))].

From (<ref>), the sequence { O(^(t)),t≥0}
is nonincreasing with O()≥0, ∀∈.
Taking the limit of T→∞ on both sides of (<ref>),
we get 

    ∑_t=1^T‖^(t)-^(t-1)‖ _2^2≤∞

 and thus

    lim_t→∞‖^(t)-^(t-1)‖ _2^2=0.



With Lemma <ref> and Lemma <ref>,
one can actually show that, given a fixed outlier thresholding ϵ>0,
the algorithm converges after finite number of iterations.




 §.§ Appendix 4: the proof of Theorem <ref>


We first show that the sequence { O(^(t)),t∈}
is bounded. It is easily to see that h_i()=‖ r_i-_i^⊺‖ _2^2
maps an unbounded set to an unbounded range. If 

    min_i=1,⋯,M{‖ r_i-_i^⊺^(τ)‖ _2^2} >ϵ,

the critic update (the updating rule in Proposition <ref>)
will stop with ^(t)=, ∀ t≥τ.
So the sequence { O(^(t)),t∈}
must be bounded such that ‖^(t)‖ _2^2≤ B,
for some B>0. 

Now ∀ t≥1, we have 

    |‖ r_i-_i^⊺^(t)‖ _2^2-‖ r_i-_i^⊺^(t-1)‖ _2^2|
    ≤    B'‖^(t)-^(t-1)‖ _2^2

for some B'>0. Then via Lemma <ref>, for a
given fixed outlier threshold parameter ϵ>0, we deduce that
there exists T∈ when t≥ T, we have 

    (‖ r_i-_i^⊺^(t)‖ _2^2-ϵ)·(‖ r_i-_i^⊺^(t-1)‖ _2^2-ϵ)≥0,

 ∀ i=1,⋯,T. That is ∀ i=1,⋯,T, u_i
remain unchanged for all t≥ T and the problem will become a
least square problem. Thus after T<∞ steps, the updating rule
in Proposition <ref> will
converge at a closed form solution

    ^(t)=(^(t-1)^⊺+ζ)^-1^(t-1).

and its corresponding ^(t).

20_home_fyzhu_DATA_Dropbox_self_Folder_myWorksO___oxs_201702_IJCAI_RobustContextualBanit_aaai


