Neural Network Reinforcement Learning for Audio-Visual Gaze Control
in Human-Robot Interaction

arXiv:1711.06834v1 [cs.RO] 18 Nov 2017

Stéphane Lathuilièrea,b , Benoit Masséa,b , Pablo Mesejoa,b , Radu Horauda,b
a Inria

Grenoble Rhône-Alpes, 655 Avenue de l’Europe, Montbonnot-Saint-Martin, 38330, France
b Univ. Grenoble Alpes, 621 avenue Centrale, Saint-Martin-d’Hères, 38400, France

Abstract
This paper introduces a novel neural network-based reinforcement learning approach for robot gaze control. Our
approach enables a robot to learn and adapt its gaze control strategy for human-robot interaction without the use
of external sensors or human supervision. The robot learns to focus its attention on groups of people from its own
audio-visual experiences, and independently of the number of people in the environment, their position and physical
appearance. In particular, we use recurrent neural networks and Q-learning to find an optimal action-selection policy,
and we pretrain on a synthetic environment that simulates sound sources and moving participants to avoid the need of
interacting with people for hours. Our experimental evaluation suggests that the proposed method is robust in terms
of parameters configuration (i.e. the selection of the parameter values has not a decisive impact on the performance).
The best results are obtained when audio and video information are jointly used, and when a late fusion strategy is
employed (i.e. when both sources of information are separately processed and then fused). Successful experiments on
a real environment with the Nao robot indicate that our framework is a step forward towards the autonomous learning
of a perceivable and socially acceptable gaze behavior.
Keywords: Reinforcement Learning, Human-Robot Interaction, Robot Gaze Control, Neural Networks, Transfer
Learning, Multimodal Data Fusion

1. Introduction

the case of the gaze control problem in a multi-person
scenario, the fact of focusing on only one person would
lead to omit important information and, therefore, to make
In recent years, there has been a growing interest in
wrong decisions. Indeed, the robot needs to follow a strathuman-robot interaction (HRI), a research field dedicated
egy to maximize useful information, and such a strategy is
to designing, evaluating and understanding robotic sysdifficult to design for two main reasons. First, the number
tems able to communicate with people [9]. The robotic
of possible configurations grows exponentially with the
agent must perceive humans and perform actions that, in
sequence length, making difficult to obtain an optimal soturn, will have an impact on the interaction. For instance,
lution for long time sequences. Second, the robot needs to
it is known that the robot’s verbal and gaze behavior has
be able to adapt its strategy to currently available data, as
a strong effect on the turn-taking conduct of the particprovided by its sensors, cameras and microphones in our
ipants [27]. Traditionally, HRI has been focused on the
case. For instance, if a companion robot enters a room
interaction between a single person with a robot. Howwith very bad acoustic conditions, the strategy needs to
ever, robots are increasingly part of groups and teams,
be adapted by decreasing the importance given to audio
e.g. performing delivery tasks in hospitals [17] or working
information.
closely alongside people on manufacturing floors [26]. In
November 21, 2017

In this paper, we consider the general problem of gaze
control, with the specific goal of finding good policies to
control the orientation of a robot head during informal
group gatherings. In particular, we propose a methodology for a robotic system to be able to autonomously learn
to focus its attention on groups of people using audiovisual information. This is a very important topic of research since perception requires not only making inferences from observations, but also making decisions about
where to look next. More specifically, we want a robot to
learn to find people in the environment, hence maximize
the number of people present in its field of view, and favor people who speak. We believe this could be useful
in many real scenarios, such as a conversation between a
humanoid robot and a group of humans, where the robot
needs to learn to look at people, in order to behave properly. The reason for using multiple sources of information
can be found in recent HRI research suggesting that no
single sensor can reliably serve to robust interaction [22].
Importantly, when it comes to the employment of multiple
sensors in complex social interactions, it becomes difficult to implement an optimal policy based on handcrafted
rules that take into consideration all possible situations
that may occur. On the contrary, we propose to follow a
data-driven approach to face such complexity. In particular, we propose to tackle this problem using a reinforcement learning (RL) approach [28]. RL is a machine learning paradigm in which agents learn by themselves by trialand-error to achieve successful strategies. As opposed to
supervised learning, there is no need for optimal decisions
at training time, only a way to evaluate how good a decision is: a reward. This paradigm, inspired by behaviorist
psychology, can allow a robot to autonomously learn a
policy that maximizes accumulated reward. In our case,
the agent, a Nao robot, autonomously moves its head depending on its knowledge about the environment. This
knowledge is called the agent’s state, and it is defined as a
sequence of audio-visual and motor observations, actions
and rewards. The optimal policy for making decisions
is learned from the reward computed using the detected
faces and the localized sound sources. The use of annotated data is not required to learn the best policy as the
agent learns autonomously by trial-and-error in an unsupervised manner. Moreover, using our approach, it is not
even necessary to make any assumption about the number of people moving in the environment or their initial

locations.
The use of RL techniques presents several advantages.
First, training using optimal decisions is not required
since the model learns from the reward obtained for each
decision taken. The reward can be considered as a feedback signal that indicates how well the robot is doing at
a given time step. Second, the robot must continuously
make judgments so as to select good actions over bad
ones. In this sense, the model can keep training at test
time and benefits from a higher adaptation ability. Finally, we avoid the need of an annotated training set or
calibration data, as our approach is unsupervised. In our
opinion, it seems entirely natural to use RL techniques
to “educate” a robot, since recent neuroscientific studies
have suggested that reinforcement affects how infants interact with their environment, including what they look at
[1], and that face looking is not innate but that environmental importance influences viewing behavior.
This paper is an extension of a recently submitted conference paper [18], where we presented the initial version of a neural network-based RL approach to address
the robot gaze control problem. In this paper, we extend
the system to deal with more complex and realistic scenarios, we delve into the impact of the main parameters
of the model, and we make a much more precise description of the whole approach. The overall proposal of this
paper can be summarized as follows:
• We propose to use recurrent neural networks, in
combination with a fully connected layer, to autonomously learn the robot gaze control strategy by
means of a value-based RL approach from multimodal data.
• We extend the visual observations to use a full
body pose detector instead of a simple face detector. While this makes both the proposed simulationbased training and online algorithms more complex,
it guarantees that more realistic scenarios are considered.
• We introduce a new algorithm to simulate moving
persons together with their respective poses. We
employ this synthetic environment to avoid the tedious training protocols that use real data, and we use
2

transfer learning from the simulated environment to
the real one.

and the time spent gazing at the robot in one interaction.
As external cameras are required, this cannot be easily applied in scenarios where the robot has to keep learning in a
real environment. Moreover, the method is limited to the
case of a single human participant. Another example of
RL applied to HRI can be found in [29] where a humanprovided reward is used to teach a robot. This idea of
interactive RL is exploited by [5] in the context of a tablecleaning robot. Visual and audio recognition are used to
get advice from a parent-like trainer to enable the robot to
learn a good policy efficiently. An extrinsic reward is used
in [25] to learn how to point a camera towards the active
speaker in a conversation. Audio information is used to
determine where to point the camera, while the reward is
provided by visual information: the active speaker raises
a blue card that can be easily identified by the robot. The
use of a multimodal DQN to learn human-like interactions is proposed both in [23] and in[24]. The robot must
choose an action (waiting, looking to a person, hand waving and hand shaking) to perform a hand shake with a human. The reward is negative if the robot tries unsuccessfully to shake hands, positive if the hand shake is successful, and null otherwise. In practice, the reward is obtained
from a sensor located in the hand of the robot and it takes
fourteen days of training to learn this skill successfully.
To the best of our knowledge, the closest work to ours is
[30] where an RL approach learns good policies to control the orientation of a mobile robot during social group
conversations. The robot learns to turn its head towards
the speaking person of the group. However, their model
is learned on simulated data that are restricted to a few
predefined scenarios with static people and a fixed spatial
organization of the group.

• We perform an extensive comparison with other neural network-based temporal architectures and evaluate the impact of the main parameters involved.
• We describe a real-time implementation of the proposed algorithm using a companion robot. The data,
the network weights, and the codes used in this paper
will be released upon acceptance of the paper 1 .

2. Related Work
RL has been studied for decades [11, 28] and has
been widely used in various topics, including robotics
[14]. Learning a policy is the main challenge in RL,
and there are two main categories of methods to address
it. First, policy-based methods define a space from the
set of policies, and sample policies from it. The reward
is then used, together with optimization techniques, e.g.
gradient-based methods, to increase the quality of subsequent sampled policies [32]. The other category, valuebased methods, consists in estimating the expected reward
for the set of possible actions. The actual policy uses this
value function to decide the suitable action, e.g. choose
the action that maximizes the value-function. In particular, popular value-based methods include Q-learning [31]
and its deep learning extension, Deep Q-Networks (or
DQNs) [20].
We now review some of the most relevant RL-based
HRI methods. In [7] an RL approach is employed to learn
a robot to play a game with a human partner. The robot
uses vision and force/torque feedback to choose the commands, and the uncertainty associated with human actions
is modeled via Gaussian processes. Bayesian optimization selects an optimal action at each time step. In [19]
RL is employed to adjust motion speed, timing, interaction distances, and gaze in the context of HRI. The reward is based on the amount of movement of the subject

In contrast to prior work, our approach allows a robot to
autonomously learn an effective gaze control policy from
audio and visual inputs in an unconstrained real-world environment. In particular, a simulated environment is used
for pretraining, thus avoiding to spend several days of
tedious real interactions with people, followed by trans1A
video
showing
additional
offline
and
onfer learning to map the learned strategies to real environline
experiments
is
already
available
at
https:
ments. Moreover, it requires neither external sensors nor
//team.inria.fr/perception/research/
neural-reinforcement-learning-for-human-robot-interaction/.
human intervention to obtain a reward.
3

Figure 1: Outline of our neural network-based reinforcement learning (RL) approach. In this example, the reward, rt−1 , includes only visual
information (Face reward). ωt , θt , and vt denote the extracted audio (Sound Source Localization, or SSL), motor and video features that compose
ot , a particular observation at time t. The observation at time t and the previous ∆t observations are used as input for the Q-Network that, as output,
provides the action, at , with the largest expected return.

3. Reinforcement Learning for Gaze Control

that determines the suitability of the action to carry out
the specified task.

3.1. Problem Formulation

The current pitch and yaw angles of the robot head are
denoted by Θt = (Θtp , Θyt ). V t ∈ {0, 1}Nv ×Mv ×J denotes the
visual observations. We consider an image of size Nv × Mv
and a multi-person 2D pose estimator returning the J joint
locations of the Nt detected persons at each time t. The
outputs of the pose estimator are pn = (unj , vnj , snj ), n ∈
[1..Nt ], j ∈ [1..J] where (unj , vnj ) ∈ Nv × Mv denotes the
pixel coordinates of the jth joints, and snj is a binary value
such that snj = 1 if the jth joints is successfully detected
and snj = 0 otherwise. We can now define the visual observations V t as follows:
(
1 i f ∃n ∈ [1..Nt ], snj = 1, (unj , vnj ) = (u, v)
V t (u, v, j) =
0 otherwise

We consider the case of a robot in a social environment.
The robot can move its head using its motors (2 degrees
of freedom) and have access to video (stereo camera) and
audio information (microphone array). We assume the existence of methods to perform body pose estimation and
sound source localization (SSL) to process video and audio data, respectively. The goal is to “teach” the robot, by
a trial-and-error learning procedure, to perform the suitable action employing those sources of information. In
this case, the suitable action corresponds to move the head
to maximize the number of people in the field of view,
favoring the attention to people who speak. The main
building blocks of our proposal are graphically displayed
in Figure 1. Throughout the paper, random variables and
their realizations are denoted by uppercase and lowercase
letters, respectively. Vectors and matrices are represented
using bold italic letters. The terms agent and robot are
used indistinctly to refer to the autonomous entity that
observes the environment and performs actions towards
achieving a particular goal.

Similarly, Ωt ∈ [0, 1]Na ×Ma denotes the audio observations, where Ωt is a heat map giving the probability that a
source is emitting a sound at each location of a grid of size
Na × Ma . It is important to notice that in practice the audio
grid is wider than the visual grid as a sound source can be
detected even if the source is outside the field of view of
the cameras. The observation variable is formally defined
The variable Ot gathers observations received at time as Ot = {Θt , V t , Ωt }, and the state variable is defined as
step t by the agent, namely visual (V t ), audio (Ωt ), and St = {O1 ...Ot } ∈ S, the set of all possible states. The robot
motor observations (Θt ). The agent uses Ot to update its can perform the action At ∈ A = {∅, ←, ↑, →, ↓}, namely
state from St−1 to St , that represents the knowledge the staying in the same position or moving its head a fixed anagent possesses about the environment. The agent then gle in one of the four cardinal directions. The reward Rt
selects an action, At , that consists on sending an input to is defined after taking action At , either as the number of
the robot motors. Finally, the agent receives a reward, Rt , faces detected in the field of view (termed Face reward in
4

section 4, and displayed in Figure 1) or as the number of
However, in practice, we employ a network Q(s, a, θ)
faces detected plus one if sound is also present in the field parametrized by weights θ to estimate the Q-function
of view (termed Speaker reward in section 4). We con- Q(s, a, θ) ≈ Q∗ (s, a) and we minimize the following loss:
sider interesting to compare a purely visual reward with a
h
i
L(θi ) = ESt , At ,Rt ,St+1 (Yi−1 − Q(St , At , θi ))2
(4)
multimodal one including audio information.
with Yi−1 = Rt + γ max(Q(St+1 , a, θi−1 )). It can been
a
seen as minimizing the mean square distance between
the approximations of the right and left hand sides of
(3). When the robot is training, we obtain a quadruplet
(St , At , Rt , St+1 ) for each time-step allowing us to compute (4). However, instead of sampling only according
to the policy implied by Q(s, a, θi ), random actions at are
taken in  percents of the time steps in order to explore
new strategies. This approach is known as epsilon-greedy
policy. L is minimized over θi+1 by stochastic gradient
descent. Refer to [21] for more technical details about the
training algorithm.

In RL, the model is learned on sequences of T states,
actions and rewards called episodes. At each time-step
t, the action at should not be chosen aiming at maximizing only the immediate reward Rt but also the future rewards (Rt+1 ...RT ). To do so, we make the standard assumption that future rewards are discounted by a
factor of γ. The parameter γ defines how much we favor rewards returned in the next coming time-steps over
longer term rewards. We then define the discounted future return R̄t as the discounted sum of future rewards
P −t τ
R̄t = Tτ=0
γ Rτ+t . We now aim at maximizing R̄t at each
time step t. In other words, the goal is to learn a policy, π(at , st ) = P(At = at |St = st ), (at , st ) ∈ A × S, such
that if the agent choose its actions according to π, the expected R̄t is maximal. The Q-function (also called actionvalue function) is defined as the expected future return
from state St taking action At and then following any policy π:

3.2. Neural Network Architectures for Q-Learning

The Q-function is modeled by a neural network that
takes as input part of the state variable St , that we define as S∆t
t = {Ot−∆t ...Ot }. The output is a vector of size
#A
that
corresponds to each Qπ (s∆t
Qπ (st , at ) = Eπ (R̄t |St = st , At = at )
(1)
t , at ), at ∈ A, where
∆t
Qπ (st , at ) is built analogously to Equation 1. Following
Learning the best policy corresponds to the following op- [21], the output layer is a Fully-Connected Layer (FCL)
timization problem Q∗ (s, a) = max[Qπ (st = s, at = a)]. with linear activations. We propose to use the Long Shortπ
The optimal Q-function obeys the identity known as the Term Memory (LSTM) [10] recurrent neural network to
model the Q-function. Batch normalization is applied to
Bellman equation:
the output of the LSTM. We argue that LSTM is wellh
i
∗
∗
Q (st , at ) = ESt+1 ,Rt Rt + γ max(Q (St+1 , a)) St = st , At = at suited for our task as it is capable of learning temporal
a
dependencies better than other recurrent neural networks
(2) and hidden Markov models. In fact, our model needs
to memorize the position and the motion of the people
This equation corresponds to the following intuition: if when it turns its head. When a person is not detected anywe have an estimator Q∗ (st , at ) for R̄t , the optimal action more, the network should be able to use previous detecat is the one that leads to the largest expected R̄t . The tions back in time in order to predict the direction towards
recursive application of this policy leads to equation (2). it should move. The J channels of V are flattened before
t
A straightforward approach would consist in updating Q the LSTM layers.
at each training step i with:
Four different network architectures are described in
this section and evaluated in the experimental section.
h
i order to evaluate when the two streams of informaIn
Qi (st , at ) = ESt+1 ,Rt Rt + γ max(Qi−1 (St+1 , a)) St = st , At = attion (audio and video) need to be fused, we propose to
a
(3) compare two strategies: early fusion and late fusion. In
5

early fusion, the unimodal features are combined into a
single representation before modeling time dependencies
(see Figure 2a, called EFNet). Conversely, in late fusion,
audio-visual features are modeled separately before fusing them (see Figure 2b, called LFNet). In order to measure the impact of each modality, we propose two more
networks using either only video (V) or only audio (Ω)
Figure 3: Unfolded representation of EFNet to better capture the sequeninformation. Figure 2c displays AudNet, the network us- tial nature of the recurrent model. Encircled crosses depict the concateing only audio information, while Figure 2d shows VisNet, nation of inputs.
that employs only visual information. Figure 2 employs a
compact representation where time t is not explicitly included, while Figure 3 depicts the unfolded representation
of EFNet where each node is associated with one particular time instance. Both figures follow the graphical representation used in [8].

Figure 4: Diagram showing all fields used in the proposed synthetic
environment. The robot’s field of view (in red) can move within the
reachable field (in blue), whereas the participants can freely move within
a larger field (in black).

two reasons. First, it would entail a long period of training, since each physical action by the robot takes an
amount of time that cannot be reduced neither by code
(a) EFNet
(b) LFNet
optimization nor by increasing our computational capabilities. Second, in the case of HRI, participants would need
to move in front of the robot for several hours or days (like
in [23]). For these two reasons, we propose to use a transfer learning approach. The Q-function is first learned on a
synthetic environment, where we simulate people moving
and talking, and it is then used to initialize the network
employed by the robot. Importantly, the network learned
from this synthetic environment can be successfully used
in the robot without the need of fine-tuning in real data.
(c) AudNet
(d) VisNet
In this synthetic environment, we do not need to generate
Figure 2: Proposed architectures to model the Q-function. Dashed lines
images and sound signals, but only the observations and
indicate connections only used in the last time step. Black squares represent a delay of a single time step. Encircled crosses depict the con- rewards the Q-Network receives as input.
catenation of inputs.

We consider that the robot can cover the field [−1, 1]2
by moving its head, but can only visually observe the
people within a small rectangular region Ft ⊂ [−1, 1]2
3.3. Pretraining on Synthetic Environment
centered in position vector Θt . The audio observations
Training from scratch a DQN model can take a long cover the whole reachable region [−1, 1]2 . However, the
time (in our case ∼150000 time steps to converge), and actual robot we use is only able to locate the yaw angle of
training directly on a robot would not be convenient for the sound sources, therefore we decided to solely provide
6

used when a person is going out vertically except that we
do not create new persons on top of the field because that
would imply the unrealistic sudden appearance of new
legs within the field. Figure 4 displays a visual representation of the different fields (or areas) defined in our synthetic environment, and Figure 5 shows an example of a
sequence of frames taken from the synthetic environment
and used during training.
Moreover, in order to favor tracking abilities, we bias
the person motion probabilities such that a person that is
faraway from the robot head orientation has a low probability to move, and a person within the field of view has
a high probability to move. Thus, when there is nobody
in the field of view, the robot cannot simply wait for a
person to come in. On the contrary, the robot needs to
track the persons that are visible. More precisely, we consider 4 different cases. First, when a person has never
been seen by the robot, the person does not move. Second, when a person is in the robot field of view (ht ∈ Ft ),
they move with a probability of 95%. Third, when the
person is further than a threshold τ ∈ R from the field of
view (||ht − Θt ||2 > τ), the probability of moving is only
25%. Finally, when the person is not visible but close to
the field of view (||ht − Θt ||2 < τ and ht < Ft ), or when the
person is unreachable (ht ∈ [−ξ, ξ]\[−1, 1]), this probability is 85%. Regarding the simulation of missing detections, we randomly ignore some faces when computing the face features. Concerning the sound modality, we
randomly choose between the following cases: 1 person
speaking, 2 persons speaking, and nobody speaking. We
use a Markov model to enforce continuity in the speaking
status of the persons, and we also simulate wrong SSL
observations.

Figure 5: Illustrative sequence taken from the synthetic environment
and employed to pretrain our neural network-based RL approach. The
moving square represents the camera field of view Ft of the robot. The
colored circles represent the joints of a person in the environment. The
large white circle represents a person speaking and, therefore, producing
speech that can be detected by the SSL system. Frames are displayed
from top to bottom and left to right.

sound observations on the horizontal axis [−1, 1]. On each
episode, we simulate one or two persons moving with random speeds and accelerations within a field [−ξ, ξ]2 where
ξ > 1. In other words, people can go to regions that are
unreachable for the robot. For each simulated person in
the current episode, we consider the position and velocity of their head at time t, ht = (uht , vht ) ∈ [−ξ, ξ]2 and
ḣ = (u̇ht , v̇ht ) ∈ R2 , respectively. At each frame, the person can keep moving, stay without moving, or choose
another random direction. The details of the simulated
environment generator are given in Algorithm 1. In a
real scenario, people can leave the scene so, in order to
simulate this phenomenon, we consider two equally probable cases when a person is going out horizontally of
the field (vht < [−ξ, ξ]). In the first case, the person is
deleted and instantly recreated on the other side of the
field (vht+1 = −vht ) keeping the same velocity (v̇ht+1 = v̇ht ).
In the second case, the person is going back towards the
center (vht+1 = vht and (v̇ht+1 = −v̇ht )). A similar approach is

From, the head position, we need to generate the position of all body joints. To do so, we propose to collect a set P of poses from an external dataset (the AVDIAR dataset [6]). We use a multiple person pose estimator on this dataset and use the detected poses for our
synthetic environment. This task is not trivial since we
need to simulate a realistic and consistent sequence of
poses. Applying tracking to the AVDIAR videos could
provide good pose sequences, but we would suffer from
three major drawbacks. First, we would have a tracking error that could affect the quality of the generated sequences. Second, each sequence would have a different
7

Data: P: a set of poses, δ: time-step
σ: velocity variance, M: pose continuity parameter

and constant size, whereas we would like to simulate sequences without size constraints. Finally, the number of
sequences would be relatively limited. In order to tackle
these three concerns, we first standardize the output coordinates obtained on AVDIAR. Considering the pose pnt
of the nth person, we sample a subset PtM ⊂ P of M
poses. Then, we select the closest pose to the current
pose: pnt+1 = argmin d(p, pnt ) where

Randomly chose N in [1..3].
for n ∈ [1..N] do
n
Initialize (hn0 , ḣ0 ) ∼
S
2
U([−1, 1]) × U([−1, −0.5] [0.5, 1])2 .
n
Randomly chose p0 in P.
p∈Π
end
   
for t ∈ [1..T − 1] do
q
u1  u2 
J
X
for n ∈ [1..N] do
1
   
j 2
j j
j
j 2
j
(s1 s2 ) (u1 − u2 ) + (v1 − v2 )
d v1  , v2  = P J
Randomly chose motion ∈ {S tay, Move}
j j
   
s
s
j=1 1 2 j=1
s2
s1
if motion = Move then
(5)
if hnt < [−ξ, ξ]2 then
The person is leaving the scene.
This distance is designed to face poses with different numSee section 3.3.
ber of detected joints. It can be interpreted as an L2 diselse
n
tance weighted by the number of visible joints in comhnt+1 ← hnt + δ( ḣt + N((0, 0), σ)).
n
n
1
mon. The intuition behind this sampling process is that
ḣt+1 ← δ (ht+1 − hnt )
when the size M of PtM increases, the probability of obend
taining a pose closer to pnt increases. Consequently, the
else
motion variability can be adjusted with the parameter M
hnt+1 ← hnt
S
n
in order to obtain a natural motion. With this method we
ḣt+1 ∼ U([−1, −0.5] [0.5, 1])2
can obtain diverse sequences of any size.
end
Draw PtM , a random set of M elements of P
pnt+1 ← argmin d(p, pnt )
4. Experiments
p∈P M
t

end
end
Algorithm 1: Generation of simulated moving poses for
our synthetic environment.

This section begins with the description of the quantitative evaluation performed on AVDIAR and synthetic
datasets. After this offline evaluation, it continues with the
description of the experiments in real time with the Nao
robot, performed to qualitatively evaluate our approach in
a real environment. Finally, the section ends with implementation details, and the results obtained and their anal- to train and test the different models on the exact same
data. In the context of RL in HRI, this is problematic beysis.
cause the data (e.g. what the robot sees and hears) depends
on
the action the robot has taken. Thus, we propose to
4.1. Offline Evaluation on the AVDIAR Dataset
first evaluate our proposal on an offline dataset. To mimic
The evaluation of HRI systems is not an easy task. the real behavior of a robot, we use the audio-visual
First, the definition of a metric to measure a correct, so- AVDIAR dataset [6]. This dataset has been recorded
cially acceptable behavior is far from trivial [33]. In our with 4 microphones and high-resolution binocular camparticular case, since gaze behavior is an important non- eras (1920 × 1080), of which we use only one. These
verbal communication cue in human-human social en- images, due to their wide field of view, are suitable to
counters [2, 12], we evaluate our approach according to simulate the whole field the robot can cover by moving
the robot capability of finding and tracking faces. Sec- its head. In practical terms, only a small part of the full
ond, in order to fairly compare different models, we need image is considered as seen by the robot.
8

4.2. Real Time Experiments on a Nao Robot

4.3. Implementation Details
In all experiments we employ the full-body pose estimator described in [4], considering the nose as the landmark that represents the face. On the Nao robot, we manage to obtain the pose in less than 100ms by selecting
carefully the research scale and downsampling the images. Considering that the Nao cameras provide images
with 10 fps, this pose estimator method can be considered
as fast enough for our scenario. Moreover, [4] has the particularity of following a bottom-up approach: each body
joint is first detected in the image, and then connected by
solving a graph matching problem. In our case, as we use
a joint heatmap, we do not need to perform this association step in order to save computation time.

In order to carry out an online evaluation of our proposal, we perform experiments on a Nao robot, developed
by Aldebaran Robotics. Nao provides a camera of 640 ×
480 pixels and four microphones. This robot is particularly well suited for HRI applications because of its design, hardware specifications and affordable cost. Nao can
detect and identify people, localize sounds, understand
some spoken words, synthesize speech and engage itself
in simple and goal-directed dialogs. Our gaze control system is implemented on top of the NAOLab middleware [3]
that synchronizes proprioceptive and perceptive information. The reason why we use a middleware is threefold.
First, the implementation is platform-independent and,
thus, easily portable. Platform-independence is crucial
since we employ a transfer learning approach to transfer
the knowledge gathered on our proposed synthetic environment to the Nao robot. Second, the use of external
computational resources is transparent. This is also a crucial matter in our case, since the full-body pose estimator
requires GPU computation for fast inference. Third, the
use of middleware makes prototyping much faster. For
all these reasons, we employed the remote and modular layer-based middleware architecture named NAOLab.
NAOLab consists of 4 layers: drivers, shared memory,
synchronization engine and application programming interface (API). Each layer is divided into 3 modules devoted to vision, audio and proprioception, respectively.
The last layer of NAOLab provides a general programming interface in C++ to handle the robot’s sensor data
and manage its actuators. NAOLab provides, at each time
step, the camera images and the yaw angle of the detected
sound sources using [15, 16].

In all scenarios we set ∆T = 4 such that each decision
is based on the last 5 observations. Different values for
∆T were tested, see Table 1, and we kept the value that
provided the best possible results without increasing the
computational complexity (in fact, values for ∆T larger
than 1 provided an almost equivalent final reward). The
output size of the LSTMs is set to 30 (since larger sizes
do not provide an improvement in performance, see Table 3), and the output size of the FCLs is set to 5 (one
per action). We use a discount factor (γ) of 0.90 (that
yields a good performance in both the AVDIAR test and
the synthetic environment, see Table 2). Concerning the
training phases, we employed the Adam optimizer [13]
and a batch size of 128. In order to help the model to
explore the policy space, we use an -greedy algorithm:
while training, a random action is chosen in % of the
cases; we decrease linearly the  value from  = 90% to
 = 10% after 150000 iterations. Concerning the observations, we employed visual and SSL heatmaps of sizes
7 × 5 for the three environments used in our experiments.
The models were trained in approximately 45 minutes on
both AVDIAR and the synthetic environment. It is interesting to notice that we obtain this training time without
using GPU, because a GPU is only required to compute
the full-body pose (in our case, a Nvidia GTX 1070 GPU).

It is important to highlight that we pretrain on the
synthetic environment before running experiments on the
Nao robot. The synthetic environment is flexible and allows us to be closer to the conditions Nao would face in
reality (field of view range, uniform location of the people
of the field, etc.). For instance, in AVDIAR, as the camera is fixed, heads are almost always at the same height.
As a consequence, the learned model would not be sufficiently general and flexible to perform well in real scenarios. Figure 5 shows a synthetic sequence employed for
pretraining our neural network-based RL system.

Concerning the details related specifically to the AVDIAR dataset, we employed 16 videos for training. The
amount of training data is doubled by flipping the video
and the SSL outputs. In order to save computation time,
the original videos are down-sampled to 1024×640 pixels.
9

2.5

2.0

2.5

2.5

LFNet
EFNet
AudNet
VisNet

2.0

LFNet
EFNet
AudNet
VisNet

0.6
2.0

1.5

1.5

1.5

1.0

1.0

1.0

0.5

0.4

0.3

0.5

0.0
0

100
time step(x1000)

150

(a) Face reward on AVDIAR
0.6

0.5

LFNet
EFNet
AudNet
VisNet

0.0
0

0.6

0.5

0.4

0.3

0.3

0.2

0.2

0.1

0.1

50

100
time step(x1000)

150

(c) Face reward on Synthetic

50

100
time step(x1000)

0.0
0

150

0.0
0

50

100
time step(x1000)

150

(a) AVDIAR

(b) Speaker reward on AVDIAR

0.4

0.0
0

0.5

0.5

50

Head Only - face_reward
Head Only - Speaker_reward
Full model- Face_reward
Full model - Speaker_reward

0.2

Head Only - face_reward
Head Only - Speaker_reward
Full model- Face_reward
Full model - Speaker_reward

0.1

0.0
0

50

100
time step(x1000)

150

(b) Synthetic

Figure 7: Evolution of the training reward obtained when using as visual
observation the result of either the full-body pose estimation or the face
detection.

LFNet
EFNet
AudNet
VisNet

4.4. Results and Discussion
50

100
time step(x1000)

In all our experiments, we run five times each model
and display the mean of five runs to lower the impact of
the stochastic training procedure. On AVDIAR, the results on both training and test sets are reported in the tables. As described previously, the synthetic environment
is randomly generated in real time, so there is no need for
a separated test set. Consequently, the mean reward over
the last 10000 time steps is reported.

150

(d) Speaker reward on Synthetic

Figure 6: Evolution of the reward obtained while training with the two
proposed rewards on the AVDIAR dataset and on the synthetic environment. We average over a 5000 time-step window for a cleaner visualization.

Table 1: Comparison of the final reward obtained using different window
lengths (∆T ). The mean and standard deviation over 5 runs are reported.
The best average results obtained are displayed in bold.

The size of the field of view where faces can be detected
is set to 300 × 200 pixels using motion steps of 36 pixels
each. At the beginning of each episode, the position of
the field of view is randomly sampled such that no face
can be seen. We noticed that this initialization procedure
favors the exploration abilities of the agent. To avoid a
bias due to the initialization procedure, we used the same
seed for all our experiments and iterated 3 times over the
10 test videos (20 when counting the flipped sequences).
An action is taken every 5 frames (0.2 seconds) and the
SSL is obtained using [6]. In the synthetic environment,
the size of field in which the people can move is set to
ξ = 1.4. In the case of Nao, the delay between two successive observations is ∼0.3 seconds. The head is free to
move in a field corresponding to 180 degrees. The motion of a single action corresponds to 0.15 radians (∼9◦ )
and 0.10 radians (∼6◦ ) for horizontal and vertical moves,
respectively.

∆T + 1
1
2
3
5
10
20

AVDIAR
Training
Test
1.75 ± 0.04 1.55 ± 0.08
1.80 ± 0.02 1.59 ± 0.03
1.82 ± 0.03 1.61 ± 0.01
1.81 ± 0.01 1.57 ± 0.03
1.80 ± 0.01 1.60 ± 0.03
1.80 ± 0.03 1.55 ± 0.05

Synthetic
0.26 ± 0.04
0.36 ± 0.04
0.42 ± 0.02
0.43 ± 0.01
0.40 ± 0.02
0.42 ± 0.02

First, we describe the experiments devoted to evaluate
the impact of some of the principal parameters involved.
Different window sizes (i.e. the number of past observations necessary to make a decision) are compared in Table 1. We can conclude that the worst results are obtained
when only the current observation is used (window size
of 1). We also observe that, on AVDIAR, the model performs well even with short window lengths (2 and 3). In
turn, with a more complex environment, as the proposed

10

synthetic environment, a longer window length tends to
perform better. We interpret that using a larger window
size helps the network to ignore the noisy observations
and to remember the position of people that left the field
of view. In Table 2, different discount factors are compared. We notice that, on AVDIAR, high discount factors
are prone to overfit as the difference in performance between training and test is higher. On the synthetic environment, low discount values perform worse because we
think that, as the environment is more complex, the model
may need several actions to reach a face. Consequently,
a model that is able to take into account the future benefit of each action performs better. Finally, in Table 3, we
compare different LSTM sizes. We observe that increasing the size does not lead to better results; an interesting
conclusion since, from a practical point of view, smaller
LSTMs are faster to train.
Table 2: Comparison of the final reward obtained using different discounted factors (γ). The mean and standard deviation over 5 runs are
reported. The best average results obtained are displayed in bold.

γ
0.25
0.50
0.75
0.90
0.99

AVDIAR
Training
Test
1.80 ± 0.47 1.60 ± 0.01
1.80 ± 0.48 1.60 ± 0.01
1.80 ± 0.49 1.54 ± 0.08
1.80 ± 0.48 1.58 ± 0.03
1.81 ± 0.49 1.55 ± 0.04

Synthetic
0.33 ± 0.09
0.35 ± 0.08
0.43 ± 0.11
0.42 ± 0.12
0.42 ± 0.12

Table 3: Comparison of the final reward obtained using different LSTM
sizes. The mean and standard deviation over 5 runs are reported. The
best average results obtained are displayed in bold.

LSTM size

AVDIAR
Training
Test
1.81 ± 0.47 1.56 ± 0.02
1.81 ± 0.50 1.60 ± 0.03
1.79 ± 0.46 1.56 ± 0.02

Synthetic

passes (specially in the experiments with the AVDIAR
dataset), meaning that the agent is learning (improving
performance) from experience. The best results are generally provided by the late fusion strategy (LFNet) and
the Speaker reward. We observe that the rewards we obtain on AVDIAR are generally higher than those obtained
on the synthetic environment. We suggest two possible
reasons. First, the synthetic environment, as described in
section 3.3, has been specifically designed to enforce exploration and tracking abilities. Consequently, it poses a
more difficult problem to solve. Second, the number of
people in AVDIAR is higher (about 4 in average), thus
finding a first person to track would be easier.
Figure 7 displays the reward obtained when using only
faces as visual observation (dashed lines) in contrast to
using the full-body pose estimation (continuous lines).
The former represents the results obtained by our previous proposal [18]. We observe that for both datasets, the
rewards are significantly higher when using the full-body
pose estimator. This figure intends to respond empirically
to the legitimate question of why a full-body pose estimator is used instead of a simple face detector. From a qualitative point of view, the answer can be found in the type of
situations that can solve one and the other. Let’s imagine
that the robot looks at the legs of a user; in case of using
only a face detector, there is no clue that could help the
robot to move up its head in order to see a face; however,
if a human full-body pose detector is used, the detection
of legs implies that there is a torso over them, and a head
over the torso. Figure 8 shows a short sequence of the AVDIAR environment, displaying the whole field covered by
the AVDIAR videos as well as the smaller field of view
captured by the robot (the red rectangle in the figure).

Finally, Table 4 shows the mean reward on the test
set for all architectures and rewards, using both AVDIAR
and synthetic data. We can notice that, on the AVDIAR
dataset using the Face reward, we obtain a mean reward
greater than 1, meaning that, on average, our model can
In Figure 6, we compare the evolution of the reward see more than one face per frame. Similarly to Figure 6,
obtained while training on the AVDIAR dataset and on higher rewards are obtained in the AVDIAR dataset, and
our synthetic environment with the two proposed rewards the best results are yielded when both modalities are taken
(Face reward and Speaker reward). Four different net- into account with LFNet. That led us to select the LFNet
works are tested: EFNet, LFNet, VisNet, and AudNet. model to perform experiments on Nao. We observe that
The y-axis of Figure 6 shows the average reward per AudNet is the worst performing approach. However, it
episode, with a clear growing trend as the training time performs quite well on AVDIAR compared to the synthetic
30
60
120

0.42 ± 0.11
0.43 ± 0.12
0.41 ± 0.10

11

Table 4: Comparison of the reward obtained with different architectures. The best results obtained are displayed in bold.

AVDIAR
Face
Network
AudNet
VisNet
EFNet
LFNet

Training
1.53 ± 0.02
1.63 ± 0.03
1.71 ± 0.04
1.81 ± 0.01

Test
1.47 ± 0.02
1.54 ± 0.04
1.53 ± 0.06
1.56 ± 0.04

Speaker
Training
Test
1.91 ± 0.03 1.84 ± 0.02
2.01 ± 0.02 1.87 ± 0.05
2.05 ± 0.02 1.85 ± 0.09
2.20 ± 0.04 1.96 ± 0.05

Face

Synthetic
Speaker

0.21 ± 0.01
0.37 ± 0.04
0.41 ± 0.03
0.42 ± 0.01

0.33 ± 0.01
0.45 ± 0.06
0.53 ± 0.03
0.52 ± 0.03

Acknowledgments

environment. This behavior can be explained by the fact
that, on AVDIAR, the SSL algorithm returns a 2D heatmap
whereas only the yaw angle is used in the synthetic environment.

Funding from the EU through the ERC Advanced Grant
VHIA #340113 is greatly acknowledged.

Concerning the experiments performed on Nao, Figure 9 shows an example of a two-person scenario using
the LFNet architecture. We managed to transfer the exploration and tracking abilities learned using the synthetic
environment. In our experiments, we see that our model
behaves well independently of the number of participants,
and the main failure cases are related to quick movements
of the participants.

References
[1] Arcaro, M.J., Schade, P.F., Vincent, J.L., Ponce,
C.R., Livingstone, M.S., 2017. Seeing faces is necessary for face-domain formation. Nature Neuroscience 20, 1404–1412.
[2] Argyle, M., 1975. Bodily communication. 1st ed.,
Routledege.

5. Conclusions

In this paper we have presented a neural network-based
reinforcement learning approach to solve the gaze robot
control problem. In particular, our agent is able to autonomously learn how to find people in the environment
by maximizing the number of people present in its field
of view (and favoring people who speak). A synthetic
environment is used for pretraining in order to perform
transfer learning to the real environment. Neither external
sensors nor human intervention are necessary to compute
the reward. Several architectures and rewards are compared on three different environments: two offline (a real
and a synthetic datasets) and one online (real time experiments using the Nao robot). Our results suggest that the
late fusion of audio and visual information represents the
best performing alternative, as well as that pretraining on
synthetic data can even make unnecessary to train on real
data.
12

[3] Badeig, F., Pelorson, Q., Arias, S., Drouard, V., Gebru, I., Li, X., Evangelidis, G., Horaud, R., 2015.
A distributed architecture for interacting with nao,
in: ACM International Conference on Multimodal
Interaction, pp. 385–386.
[4] Cao, Z., Simon, T., Wei, S.E., Sheikh, Y., 2017. Realtime Multi-Person 2D Pose Estimation using Part
Affinity Fields, in: IEEE Conference on Computer
Vision and Pattern Recognition.
[5] Cruz, F., Parisi, G.I., Twiefel, J., Wermter, S., 2016.
Multi-modal integration of dynamic audiovisual patterns for an interactive reinforcement learning scenario, in: IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 759–766.
[6] Gebru, I., Ba, S., Li, X., Horaud, R., 2017. Audiovisual speaker diarization based on spatiotemporal
bayesian fusion. IEEE Transactions on Pattern Analysis and Machine Intelligence .

Figure 8: Example of a sequence from the AVDIAR dataset. The SSL heatmap is superposed on the frame, and visible joints/limbs are displayed
using a colored skeleton. The agent’s field of view (in red) is randomly initialized (1st column), detects sound emitted by the a person and moves
accordingly (2nd). The agent manages to get all the persons in the field of view (3rd), and it looks at the three people group when two persons move
apart (4th).

Figure 9: Example of a sequence in a two-person scenario. First row shows an overview of the scene, including the participants and the robot.
Second row shows the robot’s field of view. The robot’s head is first initialized in a position where no face is visible (1st column), and the model
uses the available detections (elbow/wrist) to find the person on the right (2nd column). The robot finds the second person by looking around while
keeping the first person in its field of view (3rd column), and tracks the two people walking together (4th column).

[7] Ghadirzadeh, A., Bütepage, J., Maki, A., Kragic, D., [12] Kendon, A., 1967. Some functions of gaze-direction
Björkman, M., 2016. A sensorimotor reinforcement
in social interaction. Acta Psychologica 26, 22 – 63.
learning framework for physical Human-Robot Interaction, in: IEEE/RSJ International Conference on [13] Kingma, D.P., Ba, J., 2014. Adam: A method
for stochastic optimization, in: International ConIntelligent Robots and Systems, pp. 2682–2688.
ference on Learning Representations.
[8] Goodfellow, I., Bengio, Y., Courville, A., 2016.
[14] Kober, J., Bagnell, J.A., Peters, J., 2013. ReinforceDeep learning. MIT press.
ment learning in robotics: A survey. The Interna[9] Goodrich, M.A., Schultz, A.C., 2007. Human-robot
tional Journal of Robotics Research 32, 1238–1274.
Interaction: A Survey. Foundations and Trends in
[15] Li, X., Girin, L., Badeig, F., Horaud, R., 2016.
Human-Computer Interaction 1, 203–275.
Reverberant sound localization with a robot head
[10] Hochreiter, S., Schmidhuber, J., 1997. Long shortbased on direct-path relative transfer function, in:
term memory. Neural Computation 9, 1735–1780.
IEEE/RSJ International Conference on Intelligent
Robots and Systems, pp. 2819–2826.
[11] Kaelbling, L.P., Littman, M.L., Moore, A.W., 1996.
Reinforcement learning: A survey. Journal of artifi- [16] Li, X., Girin, L., Horaud, R., Gannot, S., 2017.
cial intelligence research 4, 237–285.
Multiple-speaker localization based on direct-path
13

features and likelihood maximization with spatial
sparsity regularization. IEEE/ACM Transactions on
Audio, Speech, and Language Processing 25, 1997–
2012.

human-robot social interaction through neural attention Q-network, in: IEEE International Conference
on Robotics and Automation, pp. 1639–1645.
[25] Rothbucher, M., Denk, C., Diepold, K., 2012.
Robotic gaze control using reinforcement learning,
in: IEEE International Workshop on Haptic Audio
Visual Environments and Games, pp. 83–88.

[17] Ljungblad, S., Kotrbova, J., Jacobsson, M., Cramer,
H., Niechwiadowicz, K., 2012. Hospital Robot
at Work: Something Alien or an Intelligent Colleague?, in: ACM Conference on Computer Supported Cooperative Work, pp. 177–186.
[18]

[19]

[20]

[21]

[26] Sauppé, A., Mutlu, B., 2015. The Social Impact of
a Robot Co-Worker in Industrial Settings, in: ACM
Conference on Human Factors in Computing SysMassé, B., Lathuilière, S., Mesejo, P., Horaud,
tems, pp. 3613–3622.
R., 2017. A reinforcement learning approach to
sensorimotor control in human-robot interaction,
[27] Skantze, G., Hjalmarsson, A., Oertel, C., 2014.
in: Submitted to IEEE International Conference on
Turn-taking, feedback and joint attention in situated
Robotics and Automation.
human-robot interaction. Speech Communication
65, 50–66.
Mitsunaga, N., Smith, C., Kanda, T., Ishiguro, H.,
Hagita, N., 2006. Robot behavior adaptation for
[28] Sutton, R.S., Barto, A.G., 1998. Introduction to Rehuman-robot interaction based on policy gradient reinforcement Learning. 1st ed., MIT Press.
inforcement learning. Journal of the Robotics Society of Japan 24, 820–829.
[29] Thomaz, A.L., Hoffman, G., Breazeal, C., 2006. Reinforcement learning with human teachers: UnderMnih, V., Kavukcuoglu, K., Silver, D., Graves, A.,
standing how people want to teach robots, in: IEEE
Antonoglou, I., Wierstra, D., Riedmiller, M., 2013.
International Symposium on Robot and Human InPlaying Atari With Deep Reinforcement Learning,
teractive Communication, pp. 352–357.
in: NIPS Deep Learning Workshop.
[30] Vázquez, M., Steinfeld, A., Hudson, S.E., 2016.
Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A.A.,
Maintaining awareness of the focus of attention of
Veness, J., Bellemare, M.G., Graves, A., Rieda conversation: A robot-centric reinforcement learnmiller, M., Fidjeland, A.K., Ostrovski, G., et al.,
ing approach, in: IEEE International Symposium on
2015. Human-level control through deep reinforceRobot and Human Interactive Communication, pp.
ment learning. Nature 518, 529–533.
36–43.

[22] Pourmehr, S., Thomas, J., Bruce, J., Wawerla, J.,
Vaughan, R., 2017. Robust sensor fusion for finding HRI partners in a crowd, in: IEEE International
Conference on Robotics and Automation, pp. 3272–
3278.

[31] Watkins, C.J.C.H., Dayan, P., 1992. Q-learning. Machine Learning 8, 279–292.
[32] Williams, R.J., 1992. Simple statistical gradientfollowing algorithms for connectionist reinforcement learning. Machine Learning .

[23] Qureshi, A.H., Nakamura, Y., Yoshikawa, Y., Ishiguro, H., 2016. Robot gains social intelligence
through multimodal deep reinforcement learning,
in: IEEE International Conference on Humanoid
Robots, pp. 745–751.

[33] Zaraki, A., Mazzei, D., Giuliani, M., Rossi, D.D.,
2014. Designing and Evaluating a Social GazeControl System for a Humanoid Robot. IEEE Transactions on Human-Machine Systems 44, 157–168.

[24] Qureshi, A.H., Nakamura, Y., Yoshikawa, Y., Ishiguro, H., 2017. Show, attend and interact: Perceivable
14

