Exact Maximum Entropy Inverse Optimal Control
for Modeling Human Attention Switching and Control*

arXiv:1607.05485v1 [cs.SY] 19 Jul 2016

Felix Schmitt1 , Hans-Joachim Bieg1 , Dietrich Manstetten1 , Michael Herman1 and Rainer Stiefelhagen2
Abstract— Maximum Causal Entropy (MCE) Inverse Optimal Control (IOC) has become an effective tool for modeling
human behavior in many control tasks. Its advantage over
classic techniques for estimating human policies is the transferability of the inferred objectives: Behavior can be predicted
in variations of the control task by policy computation using
a relaxed optimality criterion. However, exact policy inference
is often computationally intractable in control problems with
imperfect state observation. In this work, we present a model
class that allows modeling human control of two tasks of which
only one be perfectly observed at a time requiring attention
switching. We show how efficient and exact objective and
policy inference via MCE can be conducted for these control
problems. Both MCE-IOC and Maximum Causal Likelihood
(MCL)-IOC, a variant of the original MCE approach, as well
as Direct Policy Estimation (DPE) are evaluated using simulated
and real behavioral data. Prediction error and generalization
over changes in the control process are both considered in the
evaluation. The results show a clear advantage of both IOC
methods over DPE, especially in the transfer over variation
of the control process. MCE and MCL performed similar
when training on a large set of simulated data, but differed
significantly on small sets and real data.

I. INTRODUCTION
A. Motivation
Modeling and prediction of sequential human behavior
is important in many application areas. For example, [1]
recently proposed an approach to model the steering and
looking behavior of automobile drivers to predict potential
safety issues.
Realistic models typically require the collection of real-world
behavioral data. Often the control scenario’s dynamics are
already known (e.g. the vehicle model) but the policy, the
mapping from the system states to human control actions, has
to be inferred. Classically, Direct Policy Estimation (DPE)
by means of regression is employed for this purpose e.g.
as in [2]. Difficulties in the application of DPE arise for
system states that are not contained in the collected data. In
addition, small changes in the control scenario may result
in an adaptation of human behavior [3], which may pose
additional problems to DPE.
An approach to avoid the limitations of DPE is provided by
the concept of optimal policies in Markov Decision Processes
*This work was part of the public project UR:BAN which was co-funded
by the Federal Ministry for Economic Affairs and Energy on basis of a
decision by the German Bundestag.
1 F. Schmitt felix.schmitt@de.bosch.com, H.-J. Bieg, D.
Manstetten and M. Herman are with Robert Bosch GmbH, Corporate
Research, 70465 Stuttgart, Germany
2 R. Stiefelhagen is with the Institute for Anthropomatics and Robotics,
Karlsruhe Institute of Technology, Germany.

(MDP)s. Here, the policy results from maximizing reward,
which depends on the system states and agent’s control
actions (Optimal Control, OC1 ). Adaptivity in behavior is
resulting from a reward model, which determines the specific
optimal policy. For example, a static negative reward for
crashes with other cars naturally results in more cautious
driving behavior in dense traffic. Inverse Optimal Control
(IOC) can be used to infer the reward that is underlying
the observed behavior [4], [5], [6]. IOC algorithms iterate
between OC and update of the reward model until the optimal
behavior equals the observed behavior.
Unfortunately, many realistic human control problems can be
difficult to address within a classical MDP framework. First,
increased complexity due to high-dimensional state spaces
can be an issue. An important MDP class to cope with
this aspect are Linear Quadratic Regulators (LQRs). This
MDP variant is frequently used for efficient optimal control
of technical systems (e.g. lateral vehicle dynamics). System
state observability can also be an issue: For example, driving
an automobile comprises of subtasks such as monitoring the
road scenery and reading the speedometer. Because of the
limited human field of view only the states of one task at
a time can be observed and an attention2 switching strategy
is required [7], [3]. Partially Observable Markov Decision
Processes (POMDPs), an extension of MDPs, address this
issue by explicitly allowing for the fact that state information
may be imperfect [8].
OC in POMDPs can be very difficult. In most cases computationally expensive approximation techniques must be applied. The extension of LQRs to POMDPs without switching
of the agent’s attention, Linear Quadratic Gaussian (LQG)
problems, can efficiently and exactly be solved using the
Kalman filter. LQGs with switching of attention haven been
applied to human behavior [9], [10], [11]. However, OC for
attention switching in LQGs3 is considerably harder [12].
Early work solved the OC problem by exhaustive search
and only recently markable progress in solution techniques
has been made [13]. Sprague and Ballard [14] proposed
an approach to solve a nonlinear model that includes attention switching, where each subtask is solved with full
observability and a heuristic arbitration mechanism is applied
afterwards.
IOC methods have been proposed for POMDPs [15], using classical techniques for OC. IOC was also applied to
1 Depended

on the algorithms also known as Reinforcement Learning (RL)
the sense of selective attention as the prioritized allocation of limited
sensor capabilities onto a part of all objects or subtasks
3 usually referred to as measurement or sensor scheduling
2 in

model human attention switching [16]. In this approach,
the technique by [14] was used to approximate a solution.
However, both computational expensive OC and approximate
or heuristic solution can seriously undermine the advantages
of IOC over DPE in practical application: If OC is hard, this
hinders the prediction of human adaptation which requires
on-line policy re-computation. When using scenario-specific
approximate OC in iteration, IOC can fail to estimate reward
models that are transferable.
To address these issues, we present a class of LQG attention
switching problems, in which policy and rewards can be
inferred exactly. Our approach builds on the Maximum
Causal Entropy (MCE) framework [5] and extends recent
work on MCE in LQGs [17]. Although we have to restrict the
model class, we can preserve the application benefits of IOC
mentioned earlier. We have already successfully employed
this approach for modeling attention switching and driving
behavior in automobile drivers [1]. In the present article,
we present our approach in more general form and also
formulate the required procedures for computing of policies
and reward gradients. In addition, we also provide the first
performance evaluation of two IOC variants, namely MCEIOC and MCL-IOC, which showed marked differences in
performance, contrary to previous assumptions [5].
B. Contributions
We will first introduce our model class as a subclass of
POMDPs and its reduction to an MDP in section II. In
section III we review the MCE approach to IOC and its
variant Maximum Causal Likelihood IOC (MCL-IOC). Algorithms for efficient policy and reward gradient computation
for both MCE-IOC and MCL-IOC in the specific model
class are presented in section IV. Both IOC variants and
DPE are evaluated in the context of modeling driver behavior
in section V. Here, we will first report results on simulated
data before we revisit the empirical data, which were already
presented in [1]. This allows us to investigate the prediction
quality as well as the transferability to new scenarios, which
are characterized by variations in the process parameters.
II. PROBLEM FORMULATION
A. Partially Observable Markov Decision Processes
An MDP consists of a state x in an state space X , a control
or action u in a control space U, a stochastic process model
P(xt+1 |xt , ut ) and a reward function r(x, u). The objective
of an finite-horizon MDP is to find a potentially stochastic policy/controller
π(u|x) that maximizes the expected
PT
reward E[ t=0 r(xt , ut )|π, P, p0 ] over a time horizon T ,
conditioned on the policy π, the dynamics P and an initial
state distribution p0 . The optimality criterion is given by the
famous Bellman equations
QT (xT , uT ) = rT (xT , uT )

ut ∈U

Vt (xt ) = E[Qt (xt , ut )|π† (ut |xt )].

b(xt ) ∝ po (ot |xt )E[P(xt |xt−1 , ut−1 )|b(xt−1 ), ut−1 ], (4)
using the past belief b(xt−1 ) and the past applied control ut−1 . Defining states x0 = b(.)4 and a new reward
r0 (x0 ) = r0 (b(.)) := E[r(x)|b(x)], every POMDP can formally be transformed into an equivalent MDP in the belief
states [8].
B. Problem Class
We consider a control scenario, where there are two
distinct tasks that have concurring rewards rp = θp> ϕp and
rs = θs> ϕs Fig. 1.
The primary task is characterized by a process in continuous states xp , up and linear affine dynamics, that require
continuous control input. If the agent pays attention to the
primary task, the state can perfectly be observed and controls
applied accordingly. In addition to that, there is a secondary
task in discrete states and controls xs , us . The dynamics of
the secondary task are also dependent on whether the agent
pays attention to it, e.g. high reward states are only reachable
in this case. If attention is dedicated to the secondary task,
the amount of information obtained from the primary task is
reduced by a noisy observation model resulting in a decrease
in control performance. The agent’s attention is modeled by
an observation model state xo where switches by observation
models controls uo come with costs ro = θo ϕo .
Specifically, we consider the POMDPs of maximization of
T
hX
i
E
θ> ϕ(xt , ut ) P(xt+1 |xt , ut ), po (ot , |xt ), p0
t=0

(2)
(3)

xt = (xpt , xst , xot )>

(5)

ut = (upt , ust , uot )>

θ> ϕ(xt , ut ) = θp> ϕp (xpt , upt ) + θs> ϕs (xst , ust ) + θo ϕo (uot ),
were process and reward models consist of:
p
p
• A time-varying linear-affine model in x and u
xpt+1 = At xpt + Bt upt + at + p ,

(1)

Qt (xt , ut ) = rt (xt , ut ) + E[Vt+1 (xt+1 )|P(xt+1 |ut , xt )]
π† (ut |xt ) = arg max Qt (xt , ut )

Here, the function Vt (.) : X 7→ R is called Value-function
and Qt (., .) : X × U 7→ R is called Q-function. Starting with
QT (xT , uT ) the Bellman equations can in principle be used
to optimally solve MDPs recursively. However, in practical
problems X , U are usually of large size or even continuous.
This often prevents analytic computation.
POMDPs extend MDPs to states that cannot directly be
observed. Instead the agent relies on observations ot that
result from the “true” state by an Observation Model (OM)
po (ot |xt ). The observations ot can be used to maintain an
estimate of the state xt in form of a belief distribution b(.)
that encodes the agents uncertainty about the true states:
Given a new observation ot the belief is updated by

•

(6)

with system matrices At , Bt , at and normally distributed noise N (p |0, Σpn ) with noise covariance Σp .
A negative quadratic reward on xp , up :
θp> ϕp (xpt , upt ) = (xpt )> Θ1 xpt + (upt )> Θ2 upt ,

4 Note

that the belief state x0 has a continuous state space.

(7)

Secondary Task
P s (xst+1 |0; xst , ust )

Attention
xot = 0

xst

xot ?
P s (xst+1 |1; xst , ust )

(9)

under the reward θo uot , θo < 0, that implements a
switching cost.
A discrete sub-MDP of the secondary task, dependent
of the observation model state xot :
P s (xst+1 |xot ; xst , ust )

(10)

with an reward θs> ϕs (xst , ust ).

Because of the linearity in (6) and (8) the belief bt of the
partial observable state xpt is a Gaussian and fully specified
by its a-posterior expectation µpt and covariance Σpt . Hence,
these POMDPs can be transformed into belief MDPs using
the Kalman filter. Given upt , µpt , Σpt , (4) results in distributions
p(Σpt+1 |Σpt ) = Σ̂pt+1 − Σµt+1

(11)

p(µpt+1 |µpt , Σpt ) = N (µpt+1 |µµt+1 , Σµt+1 )

Kt+1 :=
Σµt+1
µµt+1

:=
:=

>
Ht+1 Σ̂pt+1 Ht+1
p
Kt+1 Ht+1 Σ̂t+1
At µpt + Bupt + at .

can substitute the covariance Σpt as well as xot . This enables
efficient computations in IV.
III. INVERSE OPTIMAL CONTROL
IOC seeks the reconstruction of the reward model r(xt , ut )
underlying observed optimal behavior. Here, the behavioral
data D is given by several sequences {(uit , xit )t=0,...,T }, i =
1, . . . , n produced by an unknown π under known initial state
p0 and process model P.
A. Reward Reconstruction Criterion
In the case of an MDP where the reward is a linear
combination of reward features r(xt , ut ) = θ> ϕ(xt , ut ),
matching the empirical feature expectation
T
n
T
hX
i
1 XX
!
ϕ(xit , uit ) = E
ϕ(xt , ut ) π† , P, p0
n i=1 t=0
t=0
|
{z
}

(17)

=:E

C. Transformation into Belief-MDP

>
Σ̂pt+1 Ht+1

p

xt

Illustration of the considered dual-task problem

mod 2

p
Σ̂pt+1 := At Σpt A>
t + Σn

p

Primary Task

with observation matrix H, and observation noise covariance Σon . xot can be switched by the OM control
uot ∈ {0, 1} according to the OM state dynamics

•

p

o t = xt

where Θ1 is negative semidefinite and Θ2 is negative
definite.
A linear Gaussian OM on the state xpt which depends
on the observation model state xot ∈ {0, 1}
(
N (ot |Hxpt , Σon ) if xot = 0
p
o
o
p (ot |xt , xt ) =
, (8)
ot = xpt
else

xot+1 = (xot + uot )

p

xt+1 = At xt + Bt ut + at + p

p

xot = 1

Fig. 1.

•

p

N (ot |Hxt , Σon )


o −1

+ Σn

(12)
(13)
(14)
(15)
(16)

The expected reward under the belief E[θp> ϕp (xpt , upt )|bt ] is
given by (µpt )> Θ1 µpt + tr(Θ1 Σpt ) + (upt )> Θ2 upt , where tr
denotes the sum of the diagonal elements of a matrix. As
the equation for Kt+1 is nonlinear in Σ̂pt+1 while uo is
discrete, the joint optimization wrt. Σ̂pt+1 , u0t is a nonlinear
mixed-integer optimization problem. This makes exact optimal measurement scheduling in linear quadratic Gaussian
problems in general hard [13]. In the class considered in this
work, however, an efficient and exact solution is possible:
As the state xpt is fully observable if xot = 1, the steps
dt ∈ {0, 1, . . . , dmax } passed since the last exact observation

by the optimal policy π† of reward θ†> ϕ(xt , ut ) suffices that
θ†> ϕ(xt , ut ) is a reward for which the observed behavior
performs equally as the optimal policy [4]. Therefore, (17)
allows to infer the reward underlying the human policy.
B. The Maximum Causal Entropy Framework
In MCE-IOC [5] condition (17) is combined with maximization of the conditional entropy −E[log π(ut |xt )|p(xt )]
of the policy under a distribution p(xt ). The distribution
p(xt ) is given by evaluation of the policy π under P and
p0 . Together this forms the optimization problem:
T
hX
i

max HP,p0 (π) := −E
log π(ut |xt ) π, P, p0
π

s.t.

t=0

T
hX
i
E
ϕ(xt , ut ) π, P, p0 = E.

(18)

t=0

As a constrained optimization problem (18) can be
solved via computation of a saddle-point (θ? , π ? ) =
arg minθ maxπ L(θ, π) of its Lagrangian function:
T
h X
i

L(π, θ) = HP,p0 (π) + θ> E
ϕ(xt , ut ) π, P, p0 − E ,
t=0

where θ is the Lagrangian multiplier of the feature matching
constraint [5]. Setting ∇π L(π, θ) ≡ 0 allows to recursively

compute π θ (ut |xt ) = arg maxπ L(θ, π) by

A. Computation of Q̃θt

Q̃θt (uT , xT ) = θ> ϕ(uT , xT ),
(19)
Z

Ṽtθ (xt+1 ) = log exp Q̃θt (ut+1 , xt+1 ) dut+1 ,


Q̃θt (ut , xt ) = θ> ϕ(ut , xt ) + E Ṽtθ (xt+1 ) P(xt+1 |ut , xt ) ,
π θ (ut |xt ) = exp(Q̃θt (ut , xt ) − Ṽtθ (xt )).

(20)

Because of the similarity of the recursion (19) with the
Bellman equations (further properties in [18] p. 76 ff.), the
resulting π θ can be interpreted as close-to-optimal policy for
the reward r(ut , xt ) = θ> ϕ(ut , xt ).
C. Reward-Learning
Classically, θ? is inferred in MCE by minimization of the
Lagrangian dual of (18), LD(θ) = maxπθ L(π θ , θ),
n X
T
i 1X
h
min OMCE (θ) := E Ṽ0θ (x0 ) p0 −
θ> ϕ(xit , uit )
θ
n i=1 t=0

by means of its gradient
n X
T
h
i 1X
θ
θ
ϕ(xit , uit ),
∇MCE
=
E
∇
Q̃
(u
,
x
)
π
,
p
−
0
θ
0 0
0 0
n i=1 t=0

[18] p. 187.
Alternatively, θ? is inferred in the MCL variant by minimization of the neg. log-likelihood of π θ (uit |xit ) (20) on data D,
min OMCL (θ) :=
θ

using its gradient
∇MCL
=
θ

n
T
i
1 XXh θ
Ṽt (xt ) − Q̃θt (ut , xt ) ,
n i=1 t=0

n
T

1 XX
E[∇Q̃θt (ut , xit )|πtθ ] − ∇Q̃θt (uit , xit )
n i=1 t=0

[19], [18] p. 189.
The gradient of Q̃θt wrt. to θ, ∇Q̃θ (ut , xt ), can, following
[19], recursively be computed by
θ
∇Q̃θt = ϕ(xt , ut ) + E[∇Q̃θt+1 |πt+1
, P, ut , xt ]
T
hX
i
=E
ϕ(xj , uj ) π θ , P, ut , xt .

(21)

We

use

(19)

for

computation of Q̃θt . Here,
can additively be split wrt.
the discrete variables uot , ust , dt , xst and the continuous
variables upt , µpt This enables to conduct the recursion by
application of the techniques of [17] for the LQG part and
the techniques of [5] for the discrete part, resulting in the
equations (23)-(25). Finally, a policy can be obtained as
Q̃θt (upt , uot , ust , µpt , dt , xst )

πtθ ∝ exp([µpt ; upt ]> Ω1t [µpt ; upt ] + [µpt ; upt ]> ωt2 )
{z
}
|

(26)

N (upt |Ft µp +ft ,ΣF
t )

· exp(τt (uot , ust , dt , xst )).

Please note, that the need of Q̃θt for establishing the stochastic MCE-policy ([6] requires Qθt ) prevents direct application
of the efficient approximation technique of [13], as it only
allows computation of the sequence of optimal switches.
B. Computation of ∇Q̃θt

Recursion (21) is used to obtain ∇Q̃θt by means of
additive splitting similar as in Q̃θt . The gradients for
the features ϕo (uot ), ϕs (ust , xst ) on the discrete variables
∇o,s Q̃θt can be obtained by enumeration as in [19]. Defin
ing Ft := [I; Ft+1 ], Tt := [At , Bt ]; [Ft+1 At , Ft+1 Bt ] and
tt := [at ; Ft+1 at + ft+1 ], the remaining gradients ∇p Q̃θt
for ϕ(upt , µpt ) are computed according to (27), (28). Here,
blk(X, Y ) is the block-diagonal matrix of X, Y , vec(X) is
the vertical concatenation of the columns of X and ⊗ the
Kronecker product.
V. EVALUATION
For empirical evaluation of both MCE- and MCL-IOC as
well as a classic DPE approach, we revisit the problem of
driver modeling already considered in [1]. Here, we seek to
model the behavior of a human who is trying to keep the
lane (primary task) while interacting with the infotainment
(secondary task) which requires glancing away from the
road.

(22)

j=t

Effectively, MCE tries to find parameters, that π θ achieves
the observed rewards. MCL instead tries to find parameters,
that π θ matches the observed airs (u, x) in the data.
MCL is more flexible than MCE as it allows extension to
simultaneous estimation of P [19], but it does not necessarily
estimate the same θ? on finite data. If D contains sufficiently
many samples that the empirical dynamics (xit+1 , uit , xit )
equal P both gradients and therefore also the estimated θs
coincide [18] p. 189. In practical application, however, one
cannot expect that this is the case. We therefore empirically
investigate the differences between MCE and MCL.
IV. INFERENCE IN PROBLEM CLASS
Similar to [17] we apply MCE to the belief MDP derived
in II-C. We will make use of the vertical and horizontal
matrix concatenation, denoted by [.; .] and [., .].

Fig. 2.

Driving while interacting with infotainment.

A. Problem Instance
We will in the following only briefly review the instance of
the problem class, for a detailed derivation and explanation
please refer to the original work [1].
The primary task (6) is modeled by states
xpt = [yt ; ẏt ; φt ; αt ], upt = α̇t and dynamics given by
the finite-time discretization of the kinematic vehicle model
  
   


ẏ
0 vt
0
y
0
0
φ̇ = 0 0 c vt  φ + 0 α̇ + −vt κt 
0 0
0
α
1
0
α̇

Q̃θT =

θ
Ṽt+1
= log
|

Z

µpT > Θ1 µpT + upT > Θ2 upT
|
{z
}

2 +ω 3
[µpT ;upT ]> Ω1T [µpT ;upT ]+[µpT ;upT ]> ωT
T

+ tr(Θ1 ΣpT (dT )) + θo uoT + θs> ϕs (xsT , usT )
|
{z
}

(23)

τT (uoT ,usT ,dT ,xsT )


2
3
+ ωt+1
+ log
exp [µpt+1 ; upt+1 ]> Ω1t+1 [µpt+1 ; upt+1 ] + [µpt+1 ; upt+1 ]> ωt+1
{z
} |
2
3
µpt+1 > Ω̄1t+1 µpt+1 +µpt+1 > ω̄t+1
+ω̄t+1

1
Ω̄1t+1 := (Ω1t+1 )µp ,µp − (Ω1t+1 )µp ,up (Ω1t+1 )−1
up ,up (Ωt+1 )up ,µp ,

Z


exp τt+1 (uot+1 , ust+1 , dt+1 , xst+1 )
{z
}
τ̄t+1 (dt+1 ,xst+1 )

2
2
2
ω̄t+1
:= (ωt+1
)µp − (Ω1t+1 )µp ,up (Ω1t+1 )−1
up ,up (ωt+1 )up

1
1 2 > 1 −1
2
3
3
)up + log(det(−Π(Ω1t+1 )−1
)up (Ωt+1 )up ,up (ωt+1
ω̄t+1
:= ωt+1
− (ωt+1
up ,up )),
4
2

(24)

Π denotes the mathematical constant

2
3
+ ω̄t+1
Q̃θt = µpt > Θ1 µpt + upt > Θ2 upt + µµt+1 > Ω̄1t+1 µµt+1 + µµt+1 > ω̄t+1
|
{z
}

(25)

[µpt ;upt ]> Ω1t [µpt ;upt ]+[µpt ;upt ]> ωt2 +ωt3

+ tr(Θ1 Σpt (dt )) + θo uot + θs> ϕs (xst , ust ) + tr(Ω̄1t+1 Σµt+1 (uot , dt )) + E[τ̄t+1 (dt+1 , xst+1 )|uot , ust , dt , xst ]
|
{z
}
τt (uot ,ust ,dt ,xst )


∇p Q̃θT = Ivec([µpT ; upT ][µpT ; upT ]> ) + 0 [µpT ; upT ] + vec blk(ΣpT (dT ), 0 )
{z
} | {z } |
|
{z
}
MT1 vec([µpT ;upT ][µpT ;upT ]> )

MT2 [µpT ;upT ]

m3T (uoT ,usT ,dT ,xsT )


2
1
1
Tt + Mt+1
(Tt ⊗ tt + tt ⊗ Tt ) [µpt ; upt ]
∇p Q̃θt = (I + Mt+1
Tt ⊗ Tt )vec([µpt ; upt ][µpt ; upt ]> ) + Mt+1
|
{z
} |
{z
}
Mt1 vec([µpt ;upt ][µpt ;upt ]> )

(27)

(28)

Mt2 [µpt ;upt ]



µ o
1
>
2
3
o
s
s
0 , ΣF
+ vec blk(Σpt (dt ), 0 ) + Mt+1
vec tt t>
t + Ft Σt (ut , dt )Ft + blk(0
t+1 ) + Mt+1 tt + E[mt+1 |πt+1 , P, ut , ut , dt , xt ]
{z
}
|
m3t (uot ,ust ,dt ,xst )

subject to a random disturbance N (p |0, Σpn ). In this context,
y is the vehicle’s lateral position wrt. lane center in m, φ
is the angle between tangent of the lane and the vehicle’s
longitudinal axis in rad, α is the steering-angle in rad, v
is the vehicle’s absolute velocity in m/s, κ is the curvature
of the lane in 1/m and c is the steering-wheel transmission ratio. Furthermore, primary task reward features (7)
ϕp (xpt , upt ) = [y 2 ; ẏ 2 ; α2 ; α̇2 ] are used. This is combined with
an observation model (8) H = [0, 0, 0, 1], Σon = 0 . For the
secondary task (10) a simple process model xst = 1 if dt > 0,
else xst = 0 with reward ϕst (xst ) = xst is applied. Summarized,
reward parameters θ := [θ1 ; θ2 ; θ3 ; θ4 ; θ5 ; θ6 ] for reward
features ϕ(xt , ut ) := [ϕp1,2,3,4 (xpt , upt ); ϕs (ust ); ϕo (uot )] have
to be inferred by IOC.
B. DPE Baseline
Given the knowledge, that the policy resulting from
both IOC approaches factorizes into π(upt |µpt )
=
o
s
N (upt |Ft µp + ft , ΣF
)
and
π(u
|d
,
x
),
we
use
the
t
t
t
t
parametrization
πb ∝ N (upt |Λb1 µp + λb2 , Σb )


· exp uot (λb3 dt + λb4 xst + λb5 (1 − xst ) ),

(26)

for comparison. Time-invariant parameters were employed,
as first experiments showed that this did not negatively affect
performance. In every experiment, Λb1 , λb2 , λb3 , λb4 , λb5 were
inferred by L1- regularized maximum likelihood estimation
for generalized linear models as implemented in MATLAB’s

lassoglm [20] using 5-fold cross-validation. Note that [1]
uses a more domain-specific model.
C. Numerical Experiments

In Pthe
numerical
experiments
a
barrier
4
−10−4 i=1 log(−θip ) was added to the objectives
OMCE,L (θ) to ensure that ΣF
t is positive definite. Both
optimization problems were solved to ≤ 10−6 relative
gradient norm.
1) Simulated Data: We first conducted an evaluation E 1
of MCE, MCL and DPE on simulated problem instances
S1 , S2 characterized by vt ≡ {500/36, 800/36} m/s, κt =
{+14, −14} × 10−4 m−1 (moderate curve on a motorway)
and with initial state xp0 = [0 m; 0 m/s; 0; 0], d0 = 0, xs0 = 0.
In both scenarios we first generated 3000 sequences
xp , xo , xs of 7s (25 Hz) by the MCE policies for θ =
[−0.5 m−2 ; −8 s2 m−2 ; −11; −200 s2 ; 0.07; −3.5]. For k =
0, 1, . . . , 10, we then selected the first 2k sequences of
instance S1 , applied DPE and estimated θ using MCE and
MCL. The original θ was used as initial guess for optimization in MCE and MCL. Thereafter the resulting policies were
used to simulate 1976 new sequences for both S1 and S2 . We
report the difference between the original state distribution
and the distribution obtained after policy inference.
The Kullback-Leibler divergence
KL(p(d)||p0 (d)) =

dmax
X
d=0

p(d)(log[p(d)] − log[p0 (d)]),

T
>

1 X
tr[(Σ0t )−1 Σ0t ] + (µt − µ0t (Σ0t )−1 (µt − µ0t
2T t=0

KLG

20.5

20.5

The results of the first evaluation E 1 on simulated data
are summarized in tables I, II. In both tables we indicated
the best result, i.e. the least median error wrt. KLG , KL or
RD, per condition by underlining. Additionally, we depict
the results of the evaluation in Fig. 3, 4 and 5.
TABLE I
E 1 : S IMULATED DATA EVALUATION
KLG
KL
20
24
28

MCE
S1
19.89
0.0901
19.29
0.0041
19.28
0.0014

S2
19.87
0.2332
19.42
0.0057
19.40
0.0009

MCL
S1
19.39
0.1219
19.27
0.0018
19.22
0.0008

S2
19.54
0.2684
19.39
0.0043
19.39
0.0006

DPE
S1
19.90
0.1645
19.35
0.1653
19.30
0.1746

S2
606.7
0.4057
686.3
0.3495
718.2
0.3560

TABLE II
E 1 : D EVIATION FROM TRUE REWARD
MCE
MCL

20
0.723
0.431

22
0.367
0.323

24
0.266
0.279

26
0.212
0.236

28
0.226
0.208

210
0.198
0.208

Similar as [4] we observed significantly5 lower prediction
error of the best IOC method compared to DPE on little
5 corresponding

to speeds {80, 90, 100, 110} km/h on the speedometer

100

101

Trajectories

102

103

Fig. 3. KLG per number of trajectories. The medians are denoted by the
line in red for MCE, in magenta for MCL and in blue for DPE. The shaded
areas indicated the [0.25, 0.75] interval for MCE, MCL and DPE. The first
plot with the continuous lines shows the results for S 1 while the second
plot with the dashed lines shows the results for S 2 .
0.4
KL

0.2
0

KL

0.4
0.2
0 0
10
Fig. 4.
meani (|θi0 − θi |/|θi |)

D. Results and Discussion

DPE

20
19.5

− dim(xpt ) + log[det(Σ0t )] − log[det(Σt )],

was used for primary task states. For MCE and MCL we also
report the mean relative deviation of the inferred θ0 from the
true θ, RD(θ, θ0 ) := meani (|θi0 − θi |/|θi |).
2) Real Data: Additionally, we evaluated on the data of
[1]. They contain 1452 snippets of 5s (25 Hz) of highway
driving at speeds {760/36, 840/36, 950/36, 1040/36} m/s
5
and varying curvature κt . The prediction quality was
assessed generating 100 sequences xp , xo , xs from the first
states in each snippet. We first conducted 10 evaluations
E 2 , where we trained on a random half of the dataset,
tested on the other and conducted the procedure reversely.
Thereafter we conducted 5 evaluations E 3 where we trained
on half of the data of a single speed and tested on the other
half and all data of other speeds. We report the median
KL(p(dt )||p0 (dt )) between the true and predicted distribution
dt and
expected squared error SE(y, y 0 ) :=
PTthe median
1
0 2
E T t=0 (yt − yt ) |π, P, p0 between the true and predicted lateral position y, y 0 over the considered snippets.
SE(y, y 0 ) was used instead of the KLG (p(xpt )||p0 (xpt )), as the
KL is not defined for the single sequence per snipped.

20
19.5

KLG

was used to compare the distribution p(d) in the original
and obtained data. The temporal mean Kullback-Leibler
divergence for Gaussians KLG (p(xpt )||p0 (xpt ))

101

Trajectories

102

103

KL per number of trajectories (Legend in Fig. 3).

1.5
1
0.5

100

Fig. 5.

101

Trajectories

102

103

RD in log-scale per number of trajectories (Legend in Fig. 3).

data. While DPE equaled to MCE on one trajectory, both
having a higher KLG than MCL, it performed significantly
worse (larger KLG ) from 24 to 128 trajectories. On more
training data, all three methods equaled in performance and
approached the average KLG , 19.15, for two sets of 1976
trajectories sampled from the true policy. When evaluating of
the unseen S2 , KLG exploded for DPE, while it only slightly
increased in both IOC methods. In KL, DPE performed
significantly 5 worse than both IOC methods in both S 1
and S 2 . MCL and MCE show differences that, however,
turned out to be statistically insignificant. Similar to KLG
DPE had a significant, yet less drastically, higher KL in S 2 .
Furthermore, DPE did not reach the low KL of the IOC
methods even for a large trainingset.
The results of the second evaluation E 2 on real data are
summarized in table III, while table IV presents the results
of the evaluation of the transfer performance E 3 .
In both evaluations E 2,3 DPE performed significantly6 worse
than both IOC methods and in KLG also worse than the
domain-specific baseline of [1] (reported value of 0.045).
MCE significantly5 outperformed MCL in KLG , while the
differences in KL turned out insignificant. In the transfer
6 all

results were tested according to Wilcoxon signed-rank test, p < 0.01

TABLE III
E 2 : REAL DATA ORDINARY EVALUATION

SE
KL

DPE
Train
0.095
0.110

Test
0.096
0.109

MCL
Train
0.021
0.072

Test
0.021
0.073

MCE
Train
0.015
0.074

R EFERENCES
Test
0.015
0.075

TABLE IV
E 3 : REAL DATA TRANSFER OVER SPEEDS

SE
KL

DPE
Same
0.088
0.098

Trans
0.097
0.118

MCL
Same
0.021
0.073

Trans
0.021
0.073

MCE
Same
0.015
0.075

Trans
0.015
0.075

to other driving speeds DPE showed a significant5 increase
in both KLG while for both IOC methods no significant
differences could be found.
As expected from the discussion of the properties of MCE
and MCL in III-C, both methods differed on little trainingdata and converged when increasing the amount of data.
As a single trajectory in the evaluation contained already
7 s × 25 Hz = 175 data, the better performance of MCL
indicates that the log-likelihood on 175 data (u, x) offers
a more robust estimation criterion than the empirical feature expectation estimated from a single trajectory. The
performance differences on real data can result from the
used linear model not capturing all physical properties of
the real vehicle. As a result, the observed policy might
systematically deviate from the shape of the MCE/MCL
policies. In that case MCE could result in lower prediction
error wrt. used metrics as it focuses stronger on matching
the state distribution than MCL (see III-C).
VI. CONCLUSION
We presented a class of human attention switching and
control problems with promising application potential.
Although computationally intractable in similar problems,
exact and efficient policy and reward inference in the MCEIOC framework is possible in this class using the proposed
recursions. Compared to DPE of the human’s policy both
investigated IOC methods, MCE and MCL, resulted in
consistently better behavior prediction. Contrary to the
expectations of previous work, however, MCL and MCE
differed significantly in evaluation: MCE showed better
results in the considered metrics and problem instances.
As the expressiveness of the conducted experiments is
limited to the problem class, future work has to investigate
the differences between MCL and MCE on other problems to
understand the specific benefits of each variant. Additionally,
it has to be investigated if ideas of [13] can also be applied
to enable approximate IOC in attention switching, e.g.
by combing it with the approximation technique of
[21]. Finally, it is worth finding out if the optimization
problem of MCE as presented in this work can be extended
to estimation of a dynamic model as proposed for MCL [19].

[1] F. Schmitt, H.-J. Bieg, D. Manstetten, M. Herman, and R. Stiefelhagen,
“Predicting lane keeping behavior of visually distracted drivers using
inverse suboptimal control,” in Proceedings of the IEEE Intelligent
Vehicles Symposium, 2016.
[2] P. Hermannstadter and B. Yang, “Driver distraction assessment using
driver modeling,” in IEEE International Conference on Systems, Man,
and Cybernetics (SMC), 2013, pp. 3693–3698.
[3] C. P. Janssen and D. P. Brumby, “Strategic adaptation to performance
objectives in a dual-task setting,” Cognitive science, vol. 34, no. 8, pp.
1548–1560, 2010.
[4] P. Abbeel and A. Y. Ng, “Apprenticeship learning via inverse reinforcement learning,” in Proceedings of the 21th International conference
on Machine learning (ICML). ACM, 2004, p. 1.
[5] B. D. Ziebart, J. Bagnell, and A. K. Dey, “Modeling interaction via
the principle of maximum causal entropy,” in Proceedings of the 27th
International Conference on Machine Learning (ICML), 2010, pp.
1255–1262.
[6] C. A. Rothkopf and C. Dimitrakakis, “Preference elicitation and
inverse reinforcement learning,” in Proceedings of the European conference on Machine learning and knowledge discovery in databases
(ECML). Springer-Verlag, 2011, pp. 34–48.
[7] J. W. Senders, A. Kristofferson, W. Levison, C. Dietrich, and J. Ward,
“The attentional demand of automobile driving,” Highway research
record, no. 195, 1967.
[8] R. D. Smallwood and E. J. Sondik, “The optimal control of partially
observable markov processes over a finite horizon,” Operations Research, vol. 21, no. 5, pp. 1071–1088, 1973.
[9] T. Sheridan, “On how often the supervisor should sample,” IEEE
Transactions on Systems Science and Cybernetics, vol. 2, no. 6, pp.
140–145, 1970.
[10] S. Baron and D. L. Kleinman, “The human as an optimal controller and
information processor,” IEEE Transactions on Man-Machine Systems,
vol. 10, no. 1, pp. 9–17, 1969.
[11] G. J. Blaauw, H. Godthelp, and P. Milgram, “Optimal control model
applications and field measurements with respect to car driving,”
Vehicle System Dynamics, vol. 13, no. 2, pp. 93–111, 1984.
[12] E. Skafidas and A. Nerode, “Optimal measurement scheduling in linear
quadratic gaussian control problems,” in Proceedings of the IEEE
International Conference on Control Applications, vol. 2, 1998, pp.
1225–1229.
[13] M. P. Vitus, W. Zhang, A. Abate, J. Hu, and C. J. Tomlin, “On efficient
sensor scheduling for linear dynamical systems,” Automatica, vol. 48,
no. 10, pp. 2482–2493, 2012.
[14] N. Sprague and D. Ballard, “Eye movements for reward maximization,” in Advances in neural information processing systems (NIPS),
2003.
[15] J. Choi and K.-E. Kim, “Inverse reinforcement learning in partially
observable environments,” The Journal of Machine Learning Research,
vol. 12, pp. 691–730, 2011.
[16] C. A. Rothkopf and D. H. Ballard, “Modular inverse reinforcement
learning for visuomotor behavior,” Biological cybernetics, vol. 107,
no. 4, pp. 477–490, 2013.
[17] X. Chen and B. D. Ziebart, “Predictive inverse optimal control for
linear-quadratic-gaussian systems,” in Proceedings of the 18th International Conference on Artificial Intelligence and Statistics (AISTATS),
2015, pp. 165–173.
[18] B. D. Ziebart, “Modeling purposeful adaptive behavior with the
principle of maximum causal entropy,” Ph.D. dissertation, Carnegie
Mellon University, Pittsburgh, PA 15213, 2010.
[19] M. Herman, T. Gindele, J. Wagner, F. Schmitt, and W. Burgard, “Inverse reinforcement learning with simultaneous estimation of rewards
and dynamics,” in Proceedings of the International Conference on
Artificial Intelligence and Statistics (AISTATS), 2016.
[20] MATLAB Statistics ToolboxTM User’s Guide, R2014b ed., The MathWorks, Inc., 3 Apple Hill Drive Natick, MA 01760-2098, 2014.
[21] M. Monfort, B. M. Lake, B. Ziebart, P. Lucey, and J. Tenenbaum,
“Softstar: Heuristic-guided probabilistic inference,” in Advances in
Neural Information Processing Systems, 2015, pp. 2746–2754.

