Learning-Based Modular Indirect Adaptive Control for a Class of
Nonlinear Systems
Mouhacine Benosman, Amir-massoud Farahmand, Meng Xia

arXiv:1509.07860v1 [] 25 Sep 2015

Abstract
We study in this paper the problem of adaptive trajectory tracking control for a class of nonlinear systems
with parametric uncertainties. We propose to use a modular approach, where we first design a robust nonlinear
state feedback which renders the closed loop input-to-state stable (ISS), where the input is considered to be the
estimation error of the uncertain parameters, and the state is considered to be the closed-loop output tracking
error. Next, we augment this robust ISS controller with a model-free learning algorithm to estimate the model
uncertainties. We implement this method with two different learning approaches. The first one is a model-free
multi-parametric extremum seeking (MES) method and the second is a Bayesian optimization-based method called
Gaussian Process Upper Confidence Bound (GP-UCB). The combination of the ISS feedback and the learning
algorithms gives a learning-based modular indirect adaptive controller. We show the efficiency of this approach
on a two-link robot manipulator example.

I. I NTRODUCTION
Classical adaptive methods can be classified into two main approaches; ‘direct approaches’, where the controller
is updated to adapt to the process, and ‘indirect approaches’, where the model is updated to better reflect the
actual process. Many adaptive methods have been proposed over the years for linear and nonlinear systems, we
could not possibly cite here all the design and analysis results that have been reported, instead we refer the
reader to e.g., [1], [2] and the references therein for more details. Of particular interest to us is the indirect
modular approach to adaptive nonlinear control, e.g., [2], [3], [4], [5], [6], [7], [8], [9], [10], [11], [12]. In this
approach, first the controller is designed by assuming that all the parameters are known and then an identifier
is used to guarantee certain boundedness of the estimation error. The identifier is independent of the designed
controller and thus the approach is called ‘modular’. For example, a modular approach has been proposed in [3]
for adaptive neural control of pure-feedback nonlinear systems, where the input-to-state stability (ISS) modularity
of the controller-estimator is achieved and the closed-loop stability is guaranteed by the small-gain theorem (see
also [13], [14]).
Mouhacine Benosman (m benosman@ieee.org) is with the Multimedia group of Mitsubishi Electric Research Laboratories (MERL),
Cambridge, MA 02139, USA. Amir-massoud Farahmand is with the Data Analytics group of MERL. Meng Xia is with the with the
Department of Electrical Engineering, University of Notre Dame, Notre Dame, IN 46556, USA (part of this work has been completed
during her internship at MERL).

In this work, we build upon this type of modular adaptive design and provide a framework which combines modelfree learning methods and robust model-based nonlinear control to propose a learning-based modular indirect
adaptive controller, where model-free learning algorithms are used to estimate, in closed-loop, the uncertain
parameters of the model. The main difference with the existing model-based indirect adaptive control methods, is
the fact that we do not use the model to design the uncertainty parameters estimation filters. Indeed, model-based
indirect adaptive controllers are based on parameters estimators designed using the system’s model, e.g., the
X-swapping methods presented in [2], where gradient descent filters obtained using the systems dynamics are
designed to estimate the uncertain parameters. We argue that because we do not use the system’s dynamics to
design uncertainties estimation filters we have less restrictions on the type of uncertainties that we can estimate,
e.g., uncertainties appearing nonlinearly can be estimated with the proposed approach, see [5] for some earlier
results on a mechatronics application. We also show here that with the proposed approach we can estimate at the
same time a vector of linearly dependent uncertainties, a case which cannot be straightforwardly solved using
model-based filters, e.g., refer to [15] where it is shown that the X-swapping model-based method fails to estimate
a vector of linearly dependent model coefficients. In this work, we implement the proposed approach with two
different model-free learning algorithms: The first one is a dither-based MES algorithm, and the second one is
a Bayesian optimization-based method called GP-UCB. The latter solves the exploration-exploitation problem in
the continuous armed bandit problem, which is a non-associative reinforcement learning (RL) setting. Indeed,
MES is a model-free control approach with well known convergence properties, since it has been analyzed in
many papers, e.g., [16], [17], [16], [18], [19]. This makes MES a good candidate for the model-free estimation
part of our modular adaptive controller, as already shown in some of our preliminary results in [7], [8], [10].
However, one of the main limitations with dither-based MES is the convergence to local minima. To improve
this part of the controller, we introduce here another model-free learning algorithm in the estimation part of the
adaptive controller. Indeed, we propose in this paper to use a reinforcement learning algorithm based on Bayesian
optimization methods, known as GP-UCB, e.g., [20], which contrary to the MES algorithm is guaranteed to reach
the global minima in a finite search space.
One point worth mentioning at this stage is that comparatively to‘pure’ model-free controllers, e.g., pure MES
or model-free RL algorithms, the proposed control here has a different goal. Indeed, the available model-free
controllers are meant for output or state regulation, i.e., solving a static optimization problem. In the contrary,
here we propose to use model-free learning to complement a model-based nonlinear control to estimate the
unknown parameters of the model, which means that the control goal, i.e., state or output trajectory tracking is
handled by the model-based controller. The learning algorithm is used to improve the tracking performance of
the model-based controller, and once the learning algorithm has converged, one can carry on using the nonlinear
model-based feedback controller alone, i.e., without the need of the learning algorithm. Furthermore, due to the

fact that we are merging together a model-based control with a model-free learning algorithm, we believe that this
type of controller can converge faster to an optimal performance, comparatively to the pure model-free controller,
since by ‘partly’ using a model-based controller, we are taking advantage of the partial information given by the
physics of the system, whereas the pure model-free algorithms assume no knowledge about the system, and thus
start the search for an optimal control signal from scratch.
Similar ideas of merging model-based control and MES has been proposed in [12], [21], [22], [4], [5], [6], [7],
[8], [10]. For instance in [12], [21] extremum seeking is used to complement a model-based controller, under
linearity of the model assumption in [12] (in the direct adaptive control setting, where the controllers gains
are estimated), or in the indirect adaptive control setting, under the assumption of linear parametrization of the
control in terms of the uncertainties in [21]. The modular design idea of using a model-based controller with ISS
guaranty, complemented with an MES-based module can be found in [5], [6], [7], [8], [10], where the MES was
used to estimate the model parameters and in [4], [23], where feedback gains were tuned using MES algorithms.
The work of this paper falls in this class of ISS-based modular indirect adaptive controllers. The difference with
other MES-based adaptive controllers is that, due to the ISS modular design we can use any model-free learning
algorithm to estimate the model uncertainties, not necessarily extremum seeking-based. To emphasis this we show
here the performance of the controller when using a type of RL-based learning algorithm as well.
The rest of the paper is organized as follows. In Section II, we present some notations, and fundamental definitions
that will be needed in the sequel. In Section III, we formulate the problem. The nominal controller design are
presented in Section IV. In Section IV-B, a robust controller is designed which guarantees ISS from the estimation
error input to the tracking error state. In Section IV-C, the ISS controller is complemented with an MES algorithm
to estimate the model parametric uncertainties. In section IV-D, we introduce the RL GP-UCB algorithm as a
model-free learning to complement the ISS controller. Section V is dedicated to an application example and the
paper conclusion is given in Section VI.
II. P RELIMINARIES
Throughout the paper, we use k · k to denote the Euclidean norm; i.e., for a vector x ∈ Rn , we have kxk ,
√
kxk2 = xT x, where xT denotes the transpose of the vector x. We denote by Card(S) the size of a finite set
qP P
n
n
2
S . The Frobenius norm of a matrix A ∈ Rm×n , with elements aij , is defined as kAkF ,
i=1
j=1 |aij | .

Given x ∈ Rm , the signum function is defined as sign(x) , [sign(x1 ), sign(x2 ), · · · , sign(xm )]T , where

sign(.) denotes the classical signum function. We use f˙ to denote the time derivative of f and f (r) (t) for the
r -th derivative of f (t), i.e. f (r) ,

dr f
dt .

We denote by Ck , functions that are k times differentiable and by C∞ , a

smooth function. A continuous function α : [0, a) → [0, ∞) is said to belong to class K if it is strictly increasing
and α(0) = 0. It is said to belong to class K∞ if a = ∞ and α(r) → ∞ as r → ∞ [24]. A continuous function
β : [0, a) × [0, ∞) → [0, ∞) is said to belong to class KL if, for a fixed s, the mapping β(r, s) belongs to class

K with respect to r and, for each fixed r , the mapping β(r, s) is decreasing with respect to s and β(r, s) → 0

as s → ∞ [24].
Next, We introduce some definitions that will be used in the sequel, e.g. [24]: Consider the system
ẋ = f (t, x, u)

(1)

where f : [0, ∞) × Rn × Rm → Rn is piecewise continuous in t and locally Lipschitz in x and u, uniformly in
t. The input u(t) is piecewise continuous, bounded function of t for all t ≥ 0.

Definition 1 ([24], [25]): The system (1) is said to be input-to-sate stable (ISS) if there exist a class KL
function β and a class K function γ such that for any initial state x(t0 ) and any bounded input u(t), the solution
x(t) exists for all t ≥ t0 and satisfies
kx(t)k ≤ β(kx(t0 )k, t − t0 ) + γ( sup ku(τ )k).
t0 ≤τ ≤t

Theorem 1 ([24], [25]): Let V : [0, ∞) × Rn → R be a continuously differentiable function such that
α1 (kxk) ≤V (t, x) ≤ α2 (kxk)

∂V
∂V
+
f (t, x, u) ≤ −W (x),
∂t
∂x

∀kxk ≥ ρ(kuk) > 0

(2)

for all (t, x, u) ∈ [0, ∞) × Rn × Rm , where α1 , α2 are class K∞ functions, ρ is a class K function, and W (x)
is a continuous positive definite function on Rn . Then, the system (1) is input-to-state stable (ISS).

Remark 1: Note that other equivalent definitions for ISS have been given in [25, pp. 1974-1975]. For instance,
Theorem 1 holds if inequality (2) is replaced by
∂V
∂V
+
f (t, x, u) ≤ −µ(kxk) + Ω(kuk)
∂t
∂x

where µ ∈ K∞

T

C 1 and Ω ∈ K∞ .

III. P ROBLEM F ORMULATION
A. Nonlinear system model
We consider here affine uncertain nonlinear systems of the form
ẋ = f (x) + ∆f (t, x) + g(x)u,

(3)

y = h(x),

where x ∈ Rn , u ∈ Rp , y ∈ Rm (p ≥ m), represent the state, the input and the controlled output vectors,
respectively. ∆f (t, x) is a vector field representing additive model uncertainties. The vector fields f , ∆f , columns
of g and function h satisfy the following assumptions.
Assumption A1 The function f : Rn → Rn and the columns of g : Rn → Rp are C∞ vector fields on a bounded

set X of Rn and h : Rn → Rm is a C∞ vector on X . The vector field ∆f (x) is C1 on X .

Assumption A2 System (3) has a well-defined (vector) relative degree {r1 , r2 , · · · , rm } at each point x0 ∈ X ,
P
and the system is linearizable, i.e., m
i=1 ri = n.
Assumption A3 The desired output trajectories yid (1 ≤ i ≤ m) are smooth functions of time, relating desired
initial points yid (0) at t = 0 to desired final points yid (tf ) at t = tf .
B. Control objectives
Our objective is to design a state feedback adaptive controller such that the output tracking error is uniformly
bounded, whereas the tracking error upper-bound is function of the uncertain parameters estimation error, which
can be decreased by the model-free learning. We stress here that the goal of learning algorithm is not stabilization
but rather performance optimization, i.e., the learning improves the parameters estimation error, which in turn
improves the output tracking error. To achieve this control objective, we proceed as follows: First, we design
a robust controller which can guarantee input-to-state stability (ISS) of the tracking error dynamics w.r.t the
estimation errors input. Then, we combine this controller with a model-free learning algorithm to iteratively
estimate the uncertain parameters, by optimizing online a desired learning cost function.
IV. A DAPTIVE C ONTROLLER D ESIGN
A. Nominal Controller
Let us first consider the system under nominal conditions, i.e., when ∆f (t, x) = 0. In this case, it is well
know, e.g., [24], that system (3) can be written as
y (r) (t) = b(ξ(t)) + A(ξ(t))u(t),

(4)

where
(r )

(r )

(rm )
y (r) (t) = [y1 1 (t), y2 2 (t), · · · , ym
(t)]T ,

ξ(t) = [ξ 1 (t), · · · , ξ m (t)]T ,
(ri −1)

ξ i (t) = [yi (t), · · · , yi

(t)],

(5)
1≤i≤m

The functions b(ξ), A(ξ) can be written as functions of f , g and h, and A(ξ) is non-singular in X̃ , where X̃
is the image of the set of X by the diffeomorphism x → ξ between the states of system (3) and the linearized
model (4). Now, to deal with the uncertain model, we first need to introduce one more assumption on system
(3).
Assumption A4 The additive uncertainties ∆f (t, x) in (3) appear as additive uncertainties in the input-output
linearized model (4)-(5) as follows (see also [26])
y (r) (t) = b(ξ(t)) + A(ξ(t))u(t) + ∆b(t, ξ(t)),

where ∆b(t, ξ) is C1 w.r.t. the state vector ξ ∈ X̃ .

(6)

Remark 2: Assumption A4, can be ensured under the so-called matching conditions ([27], p. 146).
It is well known that the nominal model (4) can be easily transformed into a linear input-output mapping.
Indeed, we can first define a virtual input vector v(t) as
v(t) = b(ξ(t)) + A(ξ(t))u(t).

(7)

Combining (4) and (7), we can obtain the following input-output mapping
y (r) (t) = v(t).

(8)

Based on the linear system (8), it is straightforward to design a stabilizing controller for the nominal system (4)
as
un = A−1 (ξ) [vs (t, ξ) − b(ξ)] ,

(9)

where vs is a m × 1 vector and the i-th (1 ≤ i ≤ m) element vsi is given by
(r )

(ri −1)

vsi = yidi − Krii (yi

(r −1)

− yidi

) − · · · − K1i (yi − yid ).

(10)

If we denote the tracking error as ei (t) , yi (t) − yid (t), we obtain the following tracking error dynamics
(ri )

ei

(t) + Krii e(ri −1) (t) + · · · + K1i ei (t) = 0,

(11)

where i ∈ {1, 2, · · · , m}. By properly selecting the gains Kji where i ∈ {1, 2, · · · , m} and j ∈
{1, 2, · · · , ri }, we can obtain global asymptotic stability of the tracking errors ei (t). To formalize this condition,

we add the following assumption.
Assumption A5 There exists a non-empty set A where Kji ∈ A such that the polynomials in (11) are Hurwitz,
where i ∈ {1, 2, · · · , m} and j ∈ {1, 2, · · · , ri }.
(ri −1)

To this end, we define z = [z 1 , z 2 , · · · , z m ]T , where z i = [ei , e˙i , · · · , ei

] and i ∈ {1, 2, · · · , m}.

Then, from (11), we can obtain
ż = Ãz,

where Ã ∈ Rn×n is a diagonal block matrix given by
Ã = diag{Ã1 , Ã2 , · · · , Ãm },

and Ãi (1 ≤ i ≤ m) is a ri × ri matrix given by

0
1


0



Ãi = 
0


..

.

−K1i
−K2i

(12)


1

..

.
1

···

···

−Krii






.





As discussed above, the gains Kji can be chosen such that the matrix Ã is Hurwitz. Thus, there exists a positive
definite matrix P > 0 such that (see e.g. [24])
ÃT P + P Ã = − I.

(13)

In the next section, we build upon the nominal controller (9) to write a robust ISS controller.
B. Lyapunov reconstruction-based ISS Controller
We now consider the uncertain model (3), i.e., when ∆f (t, x) 6= 0. The corresponding exact linearized model
is given by (6) where ∆b(t, ξ(t)) 6= 0. The global asymptotic stability of the error dynamics (11) cannot be
guaranteed anymore due to the additive uncertainty ∆b(t, ξ(t)). We use Lyapunov reconstruction techniques to
design a new controller so that the tracking error is guaranteed to be bounded given that the estimate error of
∆b(t, ξ(t)) is bounded. The new controller for the uncertain model (6) is defined as
uf = un + ur ,

(14)

where the nominal controller un is given by (9) and the robust controller ur will be given later. By using the
controller (14), and (6) we obtain
y (r) (t) = b(ξ(t)) + A(ξ(t))uf + ∆b(t, ξ(t)),
= b(ξ(t)) + A(ξ(t))un + A(ξ(t))ur + ∆b(t, ξ(t)),
= vs (t, ξ) + A(ξ(t))ur + ∆b(t, ξ(t)),

(15)

where (15) holds from (9). Which leads to the following error dynamics
ż = Ãz + B̃δ,

(16)

where Ã is defined in (12), δ is a m × 1 vector given by
δ = A(ξ(t))ur + ∆b(t, ξ(t)),

and the matrix B̃ ∈ Rn×m is given by



B̃1



 B̃2
B̃ = 
 ..
 .

B̃m

(17)






,




where each B̃i (1 ≤ i ≤ m) is given by a ri × m matrix such that

 1
for l = ri , q = i
B̃i (l, q) =
 0
otherwise.

(18)

If we choose V (z) = z T P z as a Lyapunov function for the dynamics (16), where P is the solution of the
Lyapunov equation (13), we obtain
V̇ (t) =

∂V
ż,
∂z

= z T (ÃT P + P Ã)z + 2z T P B̃δ,
= − kzk2 + 2z T P B̃δ,

(19)

where δ given by (17) depends on the robust controller ur .
Next, we design the controller ur based on the form of the uncertainties ∆b(t, ξ(t)). More specifically, we
consider here the case when ∆b(t, ξ(t)) is of the following form
∆b(t, ξ(t)) = E Q(ξ, t),

(20)

where E ∈ Rm×m is a matrix of unknown constant parameters, and Q(ξ, t) : Rn × R → Rm is a known bounded
function of sates and time variables. For notational convenience, we denote by Ê(t) the estimate of E , and by
2

eE = E − Ê , the estimate error. We define the unknown parameter vector ∆ = [E(1, 1), ..., E(m, m)]T ∈ Rm ,

b
i.e., concatenation of all elements of E , its estimate is denoted by ∆(t)
= [Ê(1, 1), ..., Ê(m, m)]T , and the

b .
estimation error vector is given by e∆ (t) = ∆ − ∆(t)
Next, we propose the following robust controller

b
ur = − A−1 (ξ)[B̃ T P zkQ(ξ, t)k2 + E(t)Q(ξ,
t)].

(21)

ż = f˜(t, z, e∆ ),

(22)

The closed-loop error dynamics can be written as

where e∆ (t) is considered to be an input to the system (22).
Theorem 2: Consider the system (3), under Assumptions A1-A5, where ∆b(t, ξ(t)) satisfies (20). If we apply
to (3) the feedback controller (14), where un is given by (9) and ur is given by (21). Then, the closed-loop
2

system (22) is ISS from the estimation errors input e∆ (t) ∈ Rm to the tracking errors state z(t) ∈ Rn .
Proof: By substitution (21) into (17), we obtain
b Q(ξ, t) + ∆b(t, ξ(t))
δ = − B̃ T P zkQ(ξ, t)k2 − E(t)
b Q(ξ, t) + E Q(ξ, t),
= − B̃ T P zkQ(ξ, t)k2 − E(t)

If we consider V (z) = z T P z as a Lyapunov function for the error dynamics (16). Then, from (19), we obtain
b Q(ξ, t)
V̇ ≤ − kzk2 + 2z T P B̃E Q(ξ, t) − 2z T P B̃ E(t)
− 2kz T P B̃k2 kQ(ξ, t)k2 ,

which leads to
V̇ ≤ − kzk2 + 2z T P B̃eE Q(ξ, t) − 2kz T P B̃k2 kQ(ξ, t)k2 .

Since z T P B̃eE Q(ξ) ≤ kz T P B̃eE Q(ξ)k ≤ kz T P B̃kkeE kF kQ(ξ)k = kz T P B̃kke∆ kkQ(ξ)k, we obtain
V̇ ≤ − kzk2 + 2kz T P B̃kke∆ kkQ(ξ, t)k − 2kz T P B̃k2 kQ(ξ, t)k2
1
1
≤ − kzk2 − 2(kz T P B̃kkQ(ξ, t)k − ke∆ k)2 + ke∆ k2
2
2
1
2
2
≤ − kzk + ke∆ k .
2

Thus, we have the following relation
1
V̇ ≤ − kzk2 ,
2

∀kzk ≥ ke∆ k > 0,

Then from the Lyapunov direct theorem in [24], [25], we obtain that system (22) is ISS from input e∆ to state
z.

C. MES-based parametric uncertainties estimation
Let us define now the following cost function
b = F (z(∆)),
b
J(∆)

(23)

where F : Rn → R, F (0) = 0, F (z) > 0 for z ∈ Rn − {0}. We need the following assumptions on J .
b ∗ = ∆.
Assumption A6 The cost function J has a local minimum at ∆

b are
Assumption A7 The initial error e∆ (t0 ) is sufficiently small, i.e., the original parameter estimate vector ∆
close enough to the actual parameter vector ∆.

Assumption A8 The cost function J is analytic and its variation with respect to the uncertain parameters is
˜ ≤ ξ2 , ξ2 > 0, ∆
˜ ∈ V(∆
b ∗ ), where V(∆
b ∗ ) denotes a compact
b ∗ , i.e., k ∂J (∆)k
bounded in the neighborhood of ∆
b
∂∆

b ∗.
neighborhood of ∆

We can now present the following result.
Lemma 3: Consider the system (3), under Assumptions A1-A8, where the uncertainty is given by (20). If we

apply to (3) the feedback controller (14), where un is given by (9), ur is given by (21), the cost function is given
b
by (23), and ∆(t)
are estimated through the ES algorithm

π
b ai > 0,
x̃˙ i = ai sin(ωi t + )J(∆),
2
b i (t) = x̃i + ai sin(ωi t − π ), i ∈ {1, 2, . . . , m2 }
∆
2

(24)

with ωi 6= ωj , ωi + ωj 6= ωk , i, j, k ∈ {1, 2, . . . , m2 }, and ωi > ω ∗ , ∀ i ∈ {1, 2, . . . , m2 }, with ω ∗ large enough.
Then, the norm of the error vector z(t) admits the following bound
kz(t)k ≤ β(kz(0)k, t) + γ(β̃(ke∆ (0)k, t) + ke∆ kmax ),

where ke∆ kmax =

ξ1
ω0

+

qP

m2 2
i=1 ai ,

ξ1 > 0, ω0 = maxi∈{1,2,...,m2 } ωi , β ∈ KL, β̃ ∈ KL and γ ∈ K.

Proof: Based on Theorem 2, we know that the tracking error dynamics (22) is ISS from the input e∆ (t) to
the state z(t). Thus, by Definition 1, there exist a class KL function β and a class K function γ such that for
any initial state z(0), any bounded input e∆ (t) and any t ≥ 0,
kz(t)k ≤ β(kz(0)k, t) + γ( sup ke∆ (τ )k).

(25)

0≤τ ≤t

b , to do so we use the results presented
Now, we need to evaluate the bound on the estimation vector ∆(t)

in [17]. First, based on Assumption A8, the cost function is locally Lipschitz, i.e. there exists η1 > 0 such
b ∗ ). Furthermore, since J is analytic, it can be
that |J(∆1 ) − J(∆2 )| ≤ η1 k∆1 − ∆2 k, for all ∆1 , ∆2 ∈ V(∆

b ∗ ) by a quadratic function, e.g. Taylor series up to the second order. Based on this
approximated locally in V(∆

and on Assumptions A6 and A7, we can obtain the following bound ([17, p. 436-437],[28])
ke∆ (t)k − kd(t)k ≤ ke∆ (t) − d(t)k ≤ β̃(ke∆ (0), tk) +

ξ1
,
ω0

where β̃ ∈ KL, ξ1 > 0, t ≥ 0, ω0 = maxi∈{1,2,...,m2 } ωi , and d(t) = [a1 sin(ω1 t + π2 ), . . . , am2 sin(ωm2 t + π2 )]T .
We can further obtain that
ξ1
+ kd(t)k
ω0
v
u m2
X
ξ1 u
a2i .
≤ β̃(ke∆ (0), tk) +
+t
ω0

ke∆ (t)k ≤ β̃(ke∆ (0), tk) +

i=1

Together with (25) yields the desired result.

Remark 3: The adaptive controller of Lemma 3 uses the ES algorithm (24) to estimate the model parametric
uncertainties. One might ask the question: where is the famous persistence of excitation (PE) condition here ? The
answer can be found in the examination of equation (24). Indeed, the ES algorithm uses as ‘input’ the sinusoidal
signals ai sin(ωi t + π2 ) which clearly satisfy the PE condition. The main difference with classical adaptive control
result, is that these excitation signals are not entering the system dynamics directly, but instead are applied as
inputs to the ES algorithm, reflected on the ES estimations outputs and thus transmitted to the system through
the feedback loop.
As we mentioned earlier, the dither-based MES has a problem of local minima, to improve this point we propose
in the next section to use GP-UCB as the model-free learning algorithm for model uncertainties estimation.

D. GP-UCB based parametric uncertainties estimation
In this section we propose to use Gaussian Process Upper Confidence Bound (GP-UCB) algorithm to find the
uncertain parameter ∆ vector [20], [29]. GP-UCB is a Bayesian optimization algorithm for stochastic optimization,
i.e., the task of finding the global optimum of an unknown function when the evaluations are potentially
contaminated with noise. The underlying working assumption for Bayesian optimization algorithms, including
GP-UCB, is that the function evaluation is costly, so we would like to minimize the number of evaluations
while having as accurate estimate of the minimizer (or maximizer) as possible [30]. For GP-UCB, this goal is
guaranteed by having an upper bound on the regret of the algorithm – to be defined precisely later.
One difficulty of stochastic optimization is that since we only observe noisy samples from the function, we
cannot really be sure about the exact value of a function at any given point. One may try to query a single
point many times in order to have an accurate estimate of the function. This, however, may lead to excessive
number of samples, and can be wasteful way of assigning samples when the true value of the function at
that point is actually far from optimal. The Upper Confidence Bounds (UCB) family of algorithms provide
a principled approach to guide the search [31]. These algorithms, which are not necessarily formulated in a
Bayesian framework, automatically balance the exploration (i.e., finding regions of the parameter space that
might be promising) and the exploration (i.e., focusing on the regions that are known to be the best based on
the current available knowledge) using the principle of optimism in the face of uncertainty. These algorithms
often come with strong theoretical guarantee about their performance. For more information about the UCB class
of algorithms, refer to [32], [33], [34]. GP-UCB is a particular UCB algorithms that is suitable to deal with
continuous domains. It uses a Gaussian Process (GP) to maintain the mean and confidence information about the
unknown function.
We briefly discuss GP-UCB in our context following the discussion of the original papers [20], [29]. Consider
the cost function J : D → R to be minimized. This function depends on the dynamics of the closed-loop system,

b used in the controller design. So we may consider it as an unknown
which itself depends on the parameters ∆

b.
function of ∆

For the moment, let us assume that J is a function sampled from a Gaussian Process (GP) [35]. Recall

that a GP is a stochastic process indexed by the set D that has the property that for any finite subset of

t
b 1, ∆
b 2, . . . , ∆
b t } ⊂ D , the joint distribution of J(∆
b i)
is a multivariate
the evaluation points, that is {∆
i=1
h
i
b = E J(∆)
b and its covariance function (or kernel)
Gaussian distribution. GP is defined by a mean function µ(∆)
h

i
b ∆
b ′ ) = Cov(J(∆),
b J(∆
b ′ )) = E J(∆)
b − µ(∆)
b
b ′ ) − µ(∆
b ′ ) . The kernel K of a GP determines the
K (∆,
J(∆
 b b′ 2 
∆k
b ∆
b ′ ) = exp − k∆−
behavior of a typical function sampled from the GP. For instance, if we choose K(∆,
,
2l2
the squared exponential kernel with length scale l > 0, it implies that the the GP is mean square differentiable

of all orders. We write J ∼ GP(µ, K ).

Let us first briefly describe how we can find the posterior distribution of a GP(0, K ); a GP with zero prior mean.
b 1, ∆
b 2, . . . , ∆
b t−1 } ⊂ D , we have observed the noisy evaluation yi = J(∆
b i ) + ηi
b t−1 , {∆
Suppose that for ∆

with ηi ∼ N (0, σ 2 ) being i.i.d. Gaussian noise. We can find the posterior mean and variance for a new point

b ∗ ∈ D as follows: Denote the vector of observed values by yt−1 = [y1 , . . . , yt−1 ]⊤ ∈ Rt−1 , and define the
∆

b i, ∆
b j ), and the vector
Grammian matrix K ∈ Rt−1×t−1 with [K]i,j = K (∆

K∗

b 1, ∆
b ∗ ), . . . , K (∆
b t−1 , ∆
b ∗ )].
= [K(∆

b ∗ ) and the variance σt (∆
b ∗ ) of the posterior of the GP evaluated at ∆
b ∗ are (cf. Section
The expected mean µt (∆

2.2 of [35])



b ∗ ) = K∗ K + σ 2 I −1 yt−1 ,
µt (∆


2 −1
b ∗ ) = K(∆
b ∗, ∆
b ∗ ) − K⊤
σt2 (∆
K∗ .
∗ K +σ I

b t by solving the following optimization
At round t, the GP-UCB algorithm selects the next query point ∆

problem1 :

b
b − β 1/2 σt−1 (∆).
b t ← argmin µt−1 (∆)
∆
t

(26)

b
∆∈D

Where βt depends on the choice of kernel among other parameters of the problem.
The optimization problem (26) is often nonlinear and non-convex. Nonetheless solving it only requires querying
the GP, which in general is much faster than querying the original dynamical system. This is important when the
dynamical system is a physical system and we would like to minimize the number of interactions with it before
b with small J(∆)
b . One practically easy way to approximately solve (26) is to restrict the search to a
finding a ∆

finite subset D ′ of D . The finite subset can be a uniform grid structure over D , or it might consist of randomly
selected members of D .
The theoretical guarantee for GP-UCB is in the form of regret upper bound. Let us define ∆∗ ← argmin∆∈D J(∆),

b t ) − J(∆∗ ). This is
the global minimizer of the objective function. The regret at time t is defined by rt = J(∆
b t according the cost function J . The cumulative regret at time T
a measure of sub-optimality of the choice of ∆
PT
is defined as RT = t=1 rt . Ideally we would like limt→∞ RTT = 0.

The behavior of the cumulative regret RT depends on the set D and the choice of kernel. If we fix the

confidence parameter δ > 0, for the squared exponential kernel, the asymptotic behavior of RT is
q

d+1
O
T [log (T ) + log(1/δ)] ,
with probability at least 1 − δ (cf. Theorem 3 of [20], [29]). This result does not even require the function J to
be a GP. It only requires the function to have a finite norm in the reproducing kernel Hilbert space (RKHS) HK
defined by the kernel K.
1

UCB algorithms are often formulated as maximization problems, so the “upper” confidence bound is calculated. Here we actually

compute the “lower” confidence bound, but to keep the naming convection, we still GP-UCB instead of GP-LCB.

Remark 4: One main difference with some of the existing model-based adaptive controllers, is the fact that
the learning-based estimation algorithm used here does not depend on the model of the system, i.e., the only
information needed to compute the learning cost function (23) is the desired trajectory and the measured output
of the system (please refer to Section V for an example). This makes the learning-based adaptive controllers
suitable for the general case of nonlinear parametric uncertainties. For example in [36], a similar preliminary
algorithm has been tested in the case of nonlinear models of electromagnetic actuators with a nonlinear parametric
uncertainty. Another point worth mentioning here, is the fact that with the available modular model-based adaptive
controllers, like the X-swapping modular algorithms, e.g., [37], it is not possible in some cases to estimate
multiple uncertainties simultaneously. For instance, it is shown in [15] that the X-swapping adaptive control
cannot estimate multiple uncertainties in the case of electromagnetic actuators, due to the linear dependency
of the uncertain parameters, i.e., when we consider three parametric uncertainties affecting the same output
acceleration, in which case the model-based estimation filters cannot distinguish between the uncertainties from
this acceleration. However, when dealing with the same example, the MES-based modular indirect adaptive control
approach was successful in estimating multiple uncertainties at the same time [28]. A similarly challenging case
is considered in the example presented in the next section.
V. T WO - LINK M ANIPULATOR E XAMPLE
We consider here a two-link robot manipulator, with the following dynamics (see e.g. [38])
H(q)q̈ + C(q, q̇)q̇ + G(q) = τ,

(27)

where q , [q1 , q2 ]T denotes the two joint angles and τ , [τ1 , τ2 ]T denotes the two joint torques. The matrix
H ∈ R4×4 is assumed to be non-singular and its elements are given by
H11 = m1 ℓ2c1 + I1 + m2 [ℓ21 + ℓ2c2 + 2ℓ1 ℓc2 cos(q2 )] + I2 ,
H12 = m2 ℓ1 ℓc2 cos(q2 ) + m2 ℓ2c2 + I2 ,

(28)

H21 = H12 ,
H22 = m2 ℓ2c2 + I2 .

The matrix C(q, q̇) is given by



C(q, q̇) , 

−hq˙2

−hq˙1 − hq˙2

hq˙1

0

where h = m2 ℓ1 ℓc2 sin(q2 ). The vector G = [G1 , G2 ]T is given by



,

G1 = m1 ℓc1 g cos(q1 ) + m2 g[ℓ2 cos(q1 + q2 ) + ℓ1 cos(q1 )],

(29)

G2 = m2 ℓc2 g cos(q1 + q2 ),

where, ℓ1 , ℓ2 are the lengths of the first and second link, respectively, ℓc1 , ℓc2 are the distances between the
rotation center and the center of mass of the first and second link respectively. m1 , m2 are the masses of the

first and second link, respectively, I1 is the moment of inertia of the first link and I2 the moment of inertia of
the second link, respectively, and g denotes the earth gravitational constant.
In our simulations, we assume that the parameters take the following values: I2 =
m2 = 5.5 kg, ℓ1 = 1.1 m, ℓ2 = 1.1 m, ℓc1 = 0.5 m, ℓc2 = 0.5 m, I1 =

11
12

5.5
12

kg · m2 , m1 = 10.5 kg,

kg · m2 , g = 9.8 m/s2 . The system

dynamics (27) can be rewritten as
q̈ = H −1 (q)τ − H −1 (q) [C(q, q̇)q̇ + G(q)] .

(30)

Thus, the nominal controller is given by
τn = [C(q, q̇)q̇ + G(q)]
+ H(q) [q¨d − Kd (q̇ − q˙d ) − Kp (q − qd )] ,

(31)

where qd = [q1d , q2d ]T , denotes the desired trajectory and the diagonal gain matrices Kp > 0, Kd > 0, are chosen
such that the linear error dynamics (as in (11)) are asymptotically stable. We choose as output references the
P
5th order polynomials q1ref (t) = q2ref (t) = 5i=0 ai (t/tf )i , where the ai ’s have been computed to satisfy the
boundary constraints qiref (0) = 0, qiref (tf ) = qf , q̇iref (0) = q̇iref (tf ) = 0, q̈iref (0) = q̈iref (tf ) = 0, i = 1, 2,

with tf = 2 sec, qf = 1.5 rad. In these tests, we assume that the nonlinear model (27) is uncertain. In particular,
we assume that there exist additive uncertainties in the model (30), i.e.,
q̈ = H −1 (q)τ − H −1 (q) [C(q, q̇)q̇ + G(q)] − E G(q).

(32)

Where, E is a matrix of constant uncertain parameters. Following (21), the robust-part of the control writes as
τr = −H(B̃ T P zkGk2 − Ê G(q)),

where



B̃ T = 

0 1 0 0
0 0 0 1

(33)



,

P is solution of the Lyapunov equation (13), with

0
1
0
0


 −Kp1 −Kd1
0
0
Ã = 

 0
0
0
1

0
0
−Kp2 −Kd2






,




z = [q1 − q1d , q̇1 − q̇1d , q2 − q2d , q̇2 − q̇2d ]T , and Ê is the matrix of the parameters’ estimates. Eventually, the

final feedback controller writes as
τ = τn + τr .

(34)

We consider here the challenging case where the uncertain parameters are linearly dependent. In this case the
uncertainties’ ‘effect’ is not observable from the measured output (see Remark 4). Indeed, in the case where the

uncertainties enter the model in a linearly dependent function, e.g. when the matric ∆ has only one non-zero line,
some of the classical available modular model-based adaptive controllers, like for instance X-swapping controllers,
cannot be used to estimate all the uncertain parameters simultaneously. For example, it has been shown in [15],
that the model-based gradient descent filters failed to estimate simultaneously multiple parameters in the case of
the electromagnetic actuators example. For instance, in comparison with the ES-based indirect adaptive controller
of [21], the modular approach does not rely on the parameters mutual exhaustive assumption, i.e., each element
of the control vector needs to be linearly dependent on at least one element of the uncertainties vector. More
specifically, we consider here the following case: ∆(1, 1) = 0.3, ∆(1, 2) = 0.6, and ∆(2, i) = 0, i = 1, 2.
In this case, the uncertainties’ effect on the acceleration q̈1 cannot be differentiated, and thus the application
of the model-based X-swapping method to estimate the actual values of both uncertainties at the same time is
challenging. Similarly, the method of [21], cannot be readily applied because the second control τ2 is not linearly
depend on the uncertainties, which only affects τ1 . However, we show next that, by using the modular ISS-based
controller, we manage to estimate the actual values of the uncertainties simultaneously and improve the tracking
performance.
A. MES-based uncertainties estimation
b i (i = 1, 2) are computed using a discrete version of (24), given by
The estimates of the two parameters ∆
π
b
xi (k + 1) = xi (k) + ai tf sin(ωi tf k + )J(∆),
2
(35)
b i (k + 1) = xi (k + 1) + ai sin(ωi tf k − π ), i = 1, 2
∆
2
b i (0) = 0. We choose the following learning cost function
where, k ∈ N denotes the iteration index, xi (0) = ∆
R
b − qd (t))T Q1 (q(∆)
b − qd (t))dt
b = tf (q(∆)
J(∆)
0
(36)
Rt
b − q̇d (t))T Q2 (q̇(∆)
b − q̇d (t))dt,
+ 0 f (q̇(∆)

where Q1 > 0 and Q2 > 0 denote the weight matrices. We implement the learning parameters: a1 = 0.1,

a2 = 0.05, ω1 = 7 rad/sec, ω2 = 5 rad/sec. The obtained performance cost function is displayed on Figure

1(a), where we see that the performance improves over the learning iterations. The corresponding parameters
estimation profiles are reported on Figures 1(b), and 1(c), which show a quick convergence of the first estimates
ˆ 1 to a neighborhood of the actual value. The convergence of the second estimates ∆
ˆ 2 is slower, which is
∆

expected from the ES algorithms when many parameters are estimated at the same time. One has to underline
here, however, that the convergence speed of the estimates and the excursion around their final mean values,
can be directly fine-tuned by the proper choice of the learning coefficients ai , ωi , i = 1, 2 in equation (35).
Finally, The tracking performance is shown on Figures 2(a), 2(b), where we can see that, after learning the actual
values of the uncertainties, the tracking of the desired trajectories is recovered. We only show the first angular
trajectories here, because the uncertainties affect directly only the acceleration q̈1 , and their effect on the tracking
for the second angular variable is negligible.

Cost Function

5
4
3
2
1
0
0

50

100

150

200

250

Number of iterations

(a) Cost function over the learning iterations (MES)
1

ˆ1
∆

0.5

0

−0.5
0

50

100

150

200

250

Number of iterations

(b) Estimate of ∆1 over the learning iterations (MES)
0.4

ˆ2
∆

0.2

0

−0.2

−0.4
0

50

100

150

200

250

Number of iterations

(c) Estimate of ∆2 over the learning iterations (MES)
Fig. 1.

Cost function and uncertainties estimates- MES algorithm

B. GP UCB-based uncertainties estimation
In this section, to show that the modular ISS-based controller is independent of the choice of the learning
algorithm, we apply the GP-UCB learning algorithm-based estimator to the same two-links manipulator example.
2

)t
We apply the algorithm IV-D, with the following parameters: σ = 0.1, l = 0.2, and βt = 2log( card(D
6δ
′

π2

), with

δ = 0.05.

We test the GP-UCB algorithm under the same conditions as in the previous section. The obtained parameters
and tracking results are reported on figures 3(a), 3(b), 3(c), 4(a), 4(b). We can see on these figures that similar

Desired and Actual q1 [rad]

2
Reference
With learning
Without learning

1.5
1
0.5
0
−0.5
0

0.5

1

1.5

2

Time [sec]

Desired and Actual q̇1 [rad/sec]

(a) Obtained vs. desired first angular trajectory (MES)

1.5
1
0.5
0
−0.5
0

0.5

1

1.5

2

Time [sec]
(b) Obtained vs. desired first angular velocity trajectory (MES)
Fig. 2.

Obtained vs. desired trajectories (MES)

to the MES-based adaptive controller, the uncertainties are well estimated. One could argue that they are better
estimated with the GP-UCB algorithm because there is no permanent dither signal, which leads to permanent
oscillations in the MES-based learning. The tracking performance is clearly improved in this case as well, due
to the precise estimation of the parameters.

6

Cost Function

5
4
3
2
1
0

50

100

150

200

Number of iterations
(a) Cost function over the learning iterations (GP-UCB)

1
0.8

ˆ1
∆

0.6
0.4
0.2
0

0

50

100

150

200

Number of iterations
(b) Estimate of ∆1 over the learning iterations (GP-UCB)

0.8

ˆ2
∆

0.6
0.4
0.2
0

50

100

150

200

Number of iterations
(c) Estimate of ∆2 over the learning iterations (GP-UCB)
Fig. 3.

Cost function and uncertainties estimates- (GP-UCB) algorithm

1.6
Reference
With learning
Without learning

Desired and Actual q1 [rad]

1.4
1.2
1
0.8
0.6
0.4
0.2
0
−0.2
0

0.5

1

1.5

2

Time [sec]

(a) Obtained vs. desired first angular trajectory (GP-UCB)
1.6

Desired and Actual q̇1 [rad/sec]

1.4
1.2
1
0.8
0.6
0.4
0.2

Reference
With learning
Without learning

0
−0.2
0

0.5

1

1.5

2

Time [sec]

(b) Obtained vs. desired first angular velocity trajectory (GPU-CB)
Fig. 4.

Obtained vs. desired trajectories (GPU-CB)

VI. C ONCLUSION
We have studied the problem of adaptive control for nonlinear systems which are affine in the control with
parametric uncertainties. For this class of systems, we have proposed the following controller: We use a modular
approach, where we first design a robust nonlinear controller, designed based on the model (assuming knowledge
of the uncertain parameters), and then complement this controller with an estimation module to estimate the actual
values of the uncertain parameters. This type of modular approaches are certainly not new, e.g., the X-swapping
methods. However, the novelty here is that the estimation module that we propose is based on model-free learning
algorithms. Indeed, we propose to use two learning algorithms, namely, a multi-parametric extremum seeking
algorithm, and a GP-UCB algorithm, to learn in realtime the uncertainties of the model. We call the learning
approach ‘model-free’ for the simple reason that it only requires to measure an output signal from the system
and compare it to a desired reference signal (independent of the model), to learn the best estimates of the

model uncertainties. We have guaranteed the stability (while learning) of the proposed approach, by ensuring
that the model-based robust controller, leads to an ISS results, which guarantees boundedness of the states of the
closed-loop system, even during the learning phase. The ISS result together with a convergent learning-algorithm
eventually leads to a bounded output tracking error, which decreases with the decrease of the estimation error. We
believe that, one of the main advantages of the proposed controller, comparatively to the existing model-based
adaptive controllers, is that we can learn (estimate) multiple uncertainties at the same time even if they appear
in the model equation in a challenging way, e.g., linearly dependent uncertainties affecting only one output, or
uncertainties appearing in a nonlinear term of the model, which are well known limitations of the model-based
estimation approaches. Another advantage of the proposed approach, is that due to its modular design, one could
easily change the learning algorithm without having to change the model-based part of the controller. Indeed, as
long as the first part of the controller, i.e., the model-based part, has been designed with a proper ISS property,
one can ‘plug into it’ any convergent learning model-free algorithm, as demonstrated here by using two different
learning approaches. We reported in this short paper some preliminary results about using GP-UCB in a modular
adaptive control setting. In a longer journal version of the work, we will report more detailed comparisons
between the MES-based adaptive controller, the GP UCB-based controller (for example in a more realistic noisy
environment), and some existing model-based ‘classical’ adaptive controllers, e.g., as found in [1].
R EFERENCES
[1] P. Ioannou and J. Sun, Robust Adaptive Control.

Dover Publications, 2012.

[2] M. Krstic, I. Kanellakopoulos, and P. Kokotovic, Nonlinear and Adpative Control Design.

New York: Wiley, 1995.

[3] C. Wang, D. J. Hill, S. S. Ge, and G. Chen, “An ISS-modular approach for adaptive neural control of pure-feedback systems,”
Automatica, vol. 42, no. 5, pp. 723–731, 2006.
[4] M. Benosman and G. Atinc, “Multi-parametric extremum seeking-based learning control for electromagnetic actuators,” in American
Control Conference, 2013, pp. 1914–1919.
[5] ——, “Nonlinear learning-based adaptive control for electromagnetic actuators,” in European Control Conference, 2013, pp. 2904–
2909.
[6] G. Atinc and M. Benosman, “Nonlinear learning-based adaptive control for electromagnetic actuators with proof of stability,” in
IEEE, Conference on Decision and Control, 2013, pp. 1277–1282.
[7] M. Benosman, “Learning-based adaptive control for nonlinear systems,” in IEEE European Control Conference, 2014.
[8] ——, “Extremum-seeking based adaptive control for nonlinear systems,” in IFAC World Congress, 2014.
[9] K. A. Wise and E. Lavretsky, “Robust and adaptive control of x-45a j-ucas: A design trade study,” in IFAC World Congress.
[10] M. Xia and M. Benosman, “Extremum seeking-based indirect adaptive control for nonlinear systems with time-varying uncertainties,”
in European Control Conference, 2015.
[11] E. Lavretsky, “Combined/composit model reference adaptive control,” IEEE, Transactions on Automatic Control, vol. 54, no. 11, pp.
2692–2697, 2009.
[12] P. Haghi and K. Ariyur, “On the extremum seeking of model reference adaptive control in higher-dimensional systems,” in American
Control Conference, 2011, pp. 1176–1181.

[13] E. D. Sontag, “Smooth stabilization implies coprime factorization,” Automatic Control, IEEE Transactions on, vol. 34, no. 4, pp.
435–443, 1989.
[14] Z.-P. Jiang, A. R. Teel, and L. Praly, “Small-gain theorem for ISS systems and applications,” Mathematics of Control, Signals and
Systems, vol. 7, no. 2, pp. 95–120, 1994.
[15] M. Benosman and G. Atinc, “Non-linear adaptive control for electromagnetic actuators,” IET Control Theory and Applications,
vol. 9, no. 2, pp. 258–269, 2015.
[16] K. B. Ariyur and M. Krstic, Real Time Optimization by Extremum Seeking Control.

New York, NY, USA: John Wiley & Sons,

Inc., 2003.
[17] M. Rotea, “Analysis of multivariable extremum seeking algorithms,” in American Control Conference, 2000. Proceedings of the
2000, vol. 1, no. 6, Sep 2000, pp. 433–437.
[18] M. Krstic and H.-H. Wang, “Stability of extremum seeking feedback for general nonlinear dynamic systems,” Automatica, vol. 36.
[19] A. R. Teel and D. Popovic, “Solving smooth and nonsmooth multivariable extremum seeking problems by the methods of nonlinear
programming,” in American Control Conference, 2001. Proceedings of the 2001, vol. 3.

IEEE, 2001, pp. 2394–2399.

[20] N. Srinivas, A. Krause, S. M. Kakade, and M. Seeger, “Gaussian process optimization in the bandit setting: No regret and experimental
design,” in Proceedings of the 27th International Conference on Machine Learning (ICML), 2010.
[21] P. Haghi and K. Ariyur, “Adpative feedback linearization of nonlinear MIMO systems using ES-MRAC,” in American Control
Conference, 2013, pp. 1828–1833.
[22] M. Benosman and G. Atinc, “Model-based learning control,” Mitsubishi Electric Research Laboratories (MERL), Patent 20130304236,
May 2012.
[23] M. Benosman, Control Theory: Perspectives, Applications and Developments, 2015, ch. Extremum Seeking-based Indirect Adaptive
Control and Feedback Gains Auto-Tuning for Nonlinear Systems.
[24] H. Khalil, Nonlinear Systems, 3rd ed.

Prentice Hall, 2002.

[25] M. Malisoff and F. Mazenc, “Further remarks on strict input-to-state stable lyapunov functions for time-varying systems,” Automatica,
vol. 41, no. 11, pp. 1973 – 1978, 2005.
[26] M. Benosman, F. Liao, K.-Y. Lum, and J. L. Wang, “Nonlinear control allocation for non-minimum phase systems,” Control Systems
Technology, IEEE Transactions on, vol. 17, no. 2, pp. 394–404, March 2009.
[27] H. Elmali and N. Olgac, “Robust output tracking control of nonlinear mimo systems via sliding mode technique,” Automatica,
vol. 28, no. 1, pp. 145–151, 1992.
[28] M. Benosman and G. Atinc, “Nonlinear backstepping learning-based adaptive control of electromagnetic actuators,” International
Journal of Control, vol. 88, no. 3, pp. 517–530, 2014.
[29] N. Srinivas, A. Krause, S. M. Kakade, and M. Seeger, “Information-theoretic regret bounds for gaussian process optimization in the
bandit setting,” IEEE Transactions on Information Theory, vol. 58, no. 5, pp. 3250–3265, 2012.
[30] E. Brochu, V. M. Cora, and N. de Freitas, “A tutorial on bayesian optimization of expensive cost functions, with application to active
user modeling and hierarchical reinforcement learning,” arXiv:1012.2599, 2010.
[31] P. Auer, N. Cesa-Bianchi, and P. Fischer, “Finite-time analysis of the multiarmed bandit problem,” Machine learning, vol. 47, no.
2-3, pp. 235–256, 2002.
[32] S. Bubeck, R. Munos, G. Stoltz, and Cs. Szepesvári, “X-armed bandits,” Journal of Machine Learning Research (JMLR), vol. 12,
no. 1655–1695, 2011.
[33] S. Bubeck and N. Cesa-Bianchi, “Regret analysis of stochastic and nonstochastic multi-armed bandit problems,” Foundations and
Trends in Machine Learning, vol. 5, no. 1, pp. 1–122, 2012.
[34] R. Munos, “From bandits to Monte-Carlo Tree Search: The optimistic principle applied to optimization and planning,” Foundations
and Trends in Machine Learning, vol. 7(1), pp. 1–130, 2014.

[35] C. E. Rasmussen and C. K. I. Williams, Gaussian Processes for Machine Learning.

MIT Press, 2006.

[36] M. Benosman, “Nonlinear learning-based adaptive control for electromagnetic actuators,” in IEEE European Control Conference,
2013, pp. 2904–2909.
[37] M. Krstic, I. Kanellakopoulos, and P. V. Kokotovic, Nonlinear and adaptive control design.

Wiley, 1995.

[38] M. W. Spong, “On the robust control of robot manipulators,” Automatic Control, IEEE Transactions on, vol. 37, no. 11, pp. 1782–
2786, 1992.

