abstract: Performing joint interaction requires constant mutual monitoring of own actions and their effects on the other's behaviour. Such an action-effect monitoring is boosted by social cues and might result in an increasing sense of agency. Joint actions and joint attention are strictly correlated and both of them contribute to the formation of a precise temporal coordination. In human-robot interaction, the robot's ability to establish joint attention with a human partner and exploit various social cues to react accordingly is a crucial step in creating communicative robots. Along the social component, an effective human-robot interaction can be seen as a new method to improve and make the robot's learning process more natural and robust for a given task. In this work we use different social skills, such as mutual gaze, gaze following, speech and human face recognition, to develop an effective teacher-learner scenario tailored to visual object learning in dynamic environments. Experiments on the iCub robot demonstrate that the system allows the robot to learn new objects through a natural interaction with a human teacher in presence of distractors.
author: Maria Lombardi$^{1*}$, Elisa Maiettini$^{1*}$, Vadim Tikhanoff$^{2}$ and Lorenzo Natale$^{1}$ [^1] [^2] [^3]
title:  **iCub Knows Where You Look: Exploiting Social Cues for Interactive Object Detection Learning**

# INTRODUCTION

Acting together to fulfil a common goal requires constant mutual monitoring of each other's actions. Such a monitoring ensures that each coactor is aware of the own actions and of their effects in the other's behaviour. The perception of the causality between the action and its effect is highly linked to the emergence of a Sense of Agency (SoA). SoA refers to the feeling of being in control of one's own action and their outcomes . It is closely related to the known phenomenon of the temporal binding (TB), largely used as implicit measure of agency. The TB effect refers to the subjective and systematic underestimation of time intervals between two related events, most commonly the action and its effect . An increase in the perceived time interval between the cause and its effect leads to a reduction in SoA.

Recent experimental developments suggest that perceiving social cues, and mainly direct eye-contact, seems to increase the TB effect . Eye contact is a fundamental social cue and plays a crucial role in non-verbal interaction between two or more individuals. The emergent literature on this topic supports the idea that eye-contact induces an enhancement of self-awareness paying more attention toward the self . Gaze represents both a social signal to understand the readiness and the attention of the partner and an implicit action to influence the partner's behaviour, clearly linking the sense of agency to the mechanism of joint attention. A crucial question is how the sense of agency may change in presence of embodied agents (and not only with other humans). Studies  assess that interacting with social robotic agents affects SoA similarly to interacting with other humans, but differently from interacting with non-social mechanical devices (e.g. air pump).

Along with the social component, endowing a robot with different social skills can lead to more unconstrained human robot interaction (HRI) pipelines in which, for instance, the robot can learn a new task by interacting with the human partner, acting as a teacher. So doing, robot's learning process may become more natural and robust to noisy environments. Specifically, in our work we enhance the iCub humanoid robot  with different human social skills in order to teach it to detect novel objects in less controlled experimental scenarios. For example, face recognition is used to recognise the teacher among other people, the ability to detect mutual gaze is needed to understand the teacher's intention, whereas gaze following is useful to understand which object the teacher wants iCub to learn. To this aim, we refer to the on-line learning approach proposed in  for object detection. Exploiting human interaction , such a method is able to train or adapt an object detection model on-line avoiding tedious waits and allowing a faster and more reactive learning process. In this work, we improve the pipelines  by relaxing some crucial constraints during the interaction, making it more natural.

Furthermore, for a successful interaction the social cues have to be accurately timed. Studies revealed that in a non-verbal context direct eye-contact should last approximately $3-4$ seconds at a time. Any longer time may cause the other person feel uncomfortable . In such studies several participants were asked to indicate their comfort level while watching different face stimuli who appeared to be looking directly at them for periods of time of variable length. Argyle and Dean proposed that there is an equilibrium point between the duration of the eye-contact and the physical proximity between the coactors. They assessed that the duration of the eye-contact increases with distance. Thus, in average for a glance of $5$ seconds long, the distance should be $60$cm . During an interaction, the duration of the eye-contact even decreases. For example, according to , when two people are having a conversation on an emotionally neutral topic the length of eye contact is $1.5$s on average.

Summarising, in our work we aim at combining the dynamics and the patterns of the social interaction with the technicalities of the object detection algorithm to reach an effective on-line collaborative learning process between iCub and the human teacher. The rest of the paper is organised as follows. In Sec. , we overview the state of art on robots having social skills and on HRI-based learning pipeline. The proposed architecture based on social learning strategy is described in Sec. . Our empirical analysis is shown in Sec.  and . Finally conclusions are drawn in Sec. .

# RELATED WORKS

## Robots as social agents

Enabling robots to exhibit social skills could lead to smoother and more effective HRI where humans and robots actually interact as peers . For example, a social robot can help a human companion in several daily activities in a better manner by predicting their intention simply observing where their attention is focused at. Humans, indeed, tend to look at an object before trying to grasp it with the hand . Gaze estimation plays a crucial role as social cue in many cooperation tasks .

Studies in social robotics identified several main characteristics that users described as factors for a robot to appear and be accepted as social . Among them, interesting factors were the capability (1) of reacting to a human's stimuli (e.g. gesture, pointing, gaze); (2) of showing emotions and feelings; (3) of being autonomous and (4) of being socially aware of the environment. Several studies exist in the current literature addressing the aforementioned points. For example,  proposed the social robot *EmIR* which was developed to provide assistance to elderly people presenting human-like features like perceiving and displaying human emotions. Specifically, they trained a convolutional neural network to detect faces on the input image and classify them according to seven classes representing specific human emotions (i.e., fear, angry, upset, happy, neutral, sad and surprised). A lot of effort has been spent in implementing socially aware robotic behaviour in terms of navigation (refer to  for a review). Among them, an extended version of the social force model for reactive navigation that integrates additional social cues used by humans while walking was proposed in . Specifically, the motion planning part of the proposed approach took into account some general social conventions, like for instance, that humans prefer to avoid each other from the left or they allow aggressive people to overtake by not coming on their way. In another work , the human can command an autonomous robot to navigate to a desired destination by raising one hand in order to trigger the robot and point to the final location with the other hand.

Humanoid robots with social abilities can be useful in various contexts. They can also provide support in the Healthcare, improving the quality of life of many people suffering of both motor impairments (e.g., after having a stroke)  and social disorders like autism . Moreover, they can be used as research tool for a better understanding of ourselves as humans  exploring social concepts like trust , empathy  and attachment .

## HRI for vision task learning

Current deep learning based methods proved to be effective for obtaining general purpose object detection models. However, gathering the sufficient ground truth for training them through supervised learning is a costly operation. It requires drawing a bounding box around each object of interest, providing the corresponding label, in each image of the training set (which, depending on the task, might be typically composed of thousands of them). One possibility to overcome this issue is to exploit the robot embodiment and its chance to interact with humans and the surrounding environment. In previous work, it has been shown that object detection methods (and specifically the one proposed in ) can be trained with human interaction  in a constrained setting. Specifically, the human shows the novel object to be trained to the robot and the information from the robot's depth sensors is used to localise the object and follow it with the robot's gaze. The latter can be segmented and the corresponding bounding box automatically assigned and gathered as ground truth. The main assumption of this approach is that the shown object is the closest element to the robot's camera. In fact, iCub takes the closest blob of pixels regardless it is the object effectively handled by the teacher. Such an issue can have a strong impact in crowded environments where other people can stand near the robot (for instance, in working places). This forces the teacher to be very close to the camera and to keep a specific behaviour during the training process, preventing a naïve user to perform it. In this work, we consider it as a starting point for our work. We endow the iCub humanoid with social skills (such as face recognition, mutual gaze detection and gaze estimation) with the aim to relax the interaction constraints considered in  and allow for a more natural interaction in the teacher-learner setting. We show with experiments that our approach make the data acquisition pipeline more robust to common causes of noise of the surrounding environment.

# PIPELINE WITH SOCIAL CUES

The proposed approach allows to exploit social cues to reduce the assumptions on the interaction required in the teacher-learner setting (Fig.  represents the considered setup). Specifically, based on social studies , we devise the learning pipeline as follows. The iCub recognises the teacher among all presents and reacts to their prolonged eye-contact ($\sim1.5$s) by engaging the interaction and waiting for verbal commands. At this point, the teacher instructs iCub on which object to learn by pronouncing its name and by showing it in their hand while looking at it. The robot, following the teacher gaze, identifies the object and starts tracking it acquiring labelled frames. Finally, iCub uses the collected data to train an object detection model. Note that, the proposed approach removes all assumptions on the shown object position with respect to the camera. The teacher can be at any distance and other elements or persons can be between the camera and the shown object. Moreover, the teacher gaze interpretation allows to eliminate any explicit trigger (e.g., microphone un-mute, hand selection) for starting parsing the user commands or identifying the hand holding the object. This permits to achieve a natural HRI, improving the downstream object detection learning.

## Pipeline overview

The proposed architecture (depicted in Fig. ) is composed of a *Detection system* and an *HRI system*. The former one implements the on-line object detection method and training pipeline of , while the latter integrates social skills (namely, face recognition, mutual gaze detection and gaze estimation) in the architecture.

The *Teacher recognition* is the first module of the proposed HRI pipeline. It endows iCub with the ability to recognise the teacher among all the people in the scene. It receives the RGB and the depth images from the robot's camera and outputs the body key-points belonging to the teacher. This module allows iCub to focus his attention on the teacher even in an environment with multiple people. The teacher's key-points are taken as input by the *Mutual gaze detection* module which recognises when the teacher wants to start an interaction with the robot by establishing eye-contact with it. Once iCub has detected the will of the teacher of starting an interaction, it focuses its attention on the teacher, waiting verbal commands. Thanks to the *Speech recogniser* module, the teacher instructs iCub on which object it is required to learn simply by speaking. The teacher body/face key-points extracted by the *Teacher recognition* are also used by the *Hand selection* module to estimate the teacher gaze direction (left or right) that allows iCub to understand which is the teacher's hand holding the object of interest. The idea behind this module exploits the concept of joint attention during interaction. It assumes that a human tends to look at the objects they are speaking about. Such a social ability allows iCub to move its gaze toward the same object the human is looking at. Exploiting such a joint attention, the *Automatic ground truth (GT) extractor* can select the blob of pixels belonging to the object held in the selected hand and start tracking it, associating it with the label given by the teacher through the verbal command. This blob of pixels, jointly with the label and the starting image, are used both as training examples by the *On-line object detection* and for tracking the object with iCub's gaze during the interaction by the *Gaze controller*. The entire behaviour is orchestrated by a *State machine*. All the aforementioned software modules with the sensors and actuators are connected through the open-source middleware YARP[^4] (Yet Another Robot Platform) . In the following paragraphs we describe how each of the main modules of the architecture was trained and built.

## Teacher recognition

The *Teacher recognition* (Fig. ) aims at recognising the teacher among all the people in the scene. Once it receives the RGB frame from the camera, OpenPose  is used to build the bounding box for each individual in the image. This is a well-known system for multi-human pose estimation. It predicts the 2D location, in pixel $(x,y)$, of the anatomical key-points of each person in the image with the corresponding confidence level $k$. Here, we extract only the face key-points in order to compute the bounding boxes and cut the input around the detected faces. The face thumbnails are given to the pre-trained network FaceNet  to extract the corresponding face embeddings ($128$-D convolutional feature vector). FaceNet is a CNN with a triplet-based loss function proposed both for face recognition and clustering (we use it to extract the feature vector for each face). Then, the feature vectors are given to a binary Support Vector Machine (SVM) classifier with RBF kernel trained to recognise the teacher from other people in the scene. The hyperparameters of the SVM model were selected using a randomised search over a grid parameters. The module's output is the OpenPose skeleton that belongs to the teacher.

This module works completely online. It allows to acquire batches of sample data and train a new SVM model as the data increases (one batch has $300$ samples for each class). The samples (image crops of the face extracted with FaceNet as described above) are automatically labelled as *teacher* whereas the samples belonging to the negative class are labelled randomly picking data from the LFW dataset[^5]. Such a dataset contains more than $6$k labelled faces and is the same used to train FaceNet. The incremental batches of data acquired online are randomly split in train and validation set (the $30\%$ of the dataset is used as validation set). The training ends when the accuracy on the validation set reaches a threshold of $0.99$. After that, the module uses the last saved model to produce its output. The main advantage to have a face recogniser that can be trained online is that the system is not constrained to a specific subset of teachers pre-trained a-priori. Furthermore, to allow this module to efficiently work online in a real training scenario, a tracking system of the teacher's body has been implemented in case their face is not visible from the camera (for example, when iCub turns its head to track the object of interest). Specifically, we track the keypoint representing the teacher's hip comparing the relative distance between it and that of all the people in the scene in consecutive frames.

## Mutual gaze detection

For this module we rely on . Briefly, such a classifier exploits OpenPose to extract the feature vector of the individual of interest (in the proposed application this is identified by the *Teacher recognition* module). A subset of $19$ face key-points are considered ($8$ points for each eye, $2$ points for the ears and $1$ for the nose) resulting in a feature vector of $57$ elements (i.e. the triplet $(x,y,k)$ is taken for each point). Then, a binary SVM classifier is trained to recognise events of *eye-contact / no eye-contact*. For details, refer to .

## Hand selection

The *Hand selection* module was trained using the RGB frames collected in  with the iCub cameras. This dataset is composed of images from $24$ participants that move their gaze in different directions. In this work, we manually labelled with *left / right* each frame and we used them to train a gaze direction estimation model, that discriminating the teacher gaze direction, allows the robot to focus on the hand a human is looking at.

Each RGB frame acquired from the camera is given as input to OpenPose in order to extract the feature vector. As for the *Mutual gaze detection*, we consider the $19$ teacher face key-points as feature vector. This feature vector is centred with the respect to the head centroid and normalised on the farthest point from it. The resulting feature vector is finally used to train a two-classes SVM with RBF kernel. Specifically, the hyperparameters of the SVM model are selected using a 5-fold grid search cross-validation. For the training of the classifier, we split the dataset such that $19$ out of the total $24$ participants are used as training set, while the remaining $5$ as test set for performance evaluation. We repeat the split for $5$ trials in order to average the performance over different subsets of participants. The reported performances over the test sets are: accuracy $= 0.99 \pm 0.01$, precision $= 0.98 \pm 0.02$, recall $= 1.00$, F1-score $= 0.99 \pm 0.01$. At inference time, the module's output is the pair $(p,c)$ where $p$ is the prediction of the hand the human is looking at, while $c \in [0,1]$ is the confidence level.

## On-line object detection

For this module we rely on the method presented in . Given an input image, it consists of (i) a first stage of region proposals and feature extraction and (ii) a second stage of region classification and bounding box refinement. The former relies on layers from Mask R-CNN . They are used to extract a set of Regions of Interest (RoIs) from the input image and encode them into convolutional features. In this work, we considered ResNet50  as the convolutional backbone for Mask R-CNN. The second part classifies and refines the RoIs proposed at the previous stage by using respectively a set of FALKON  binary classifiers (one for each object class to detect) for the classification and Regularized Least Squares (RLS) for the refinement. Specifically, the training of the classifiers applies an approximated bootstrapping approach, called Minibootstrap  to overcome the background-foreground class imbalance problem in object detection, keeping a fast learning. Refer to  for details.

## Automatic GT extractor

The main drawback of previous work on HRI systems for robot learning, like e.g. , is the list of assumptions regarding the human teacher. In , for instance, in the teacher-learner setting, the main hypothesis is that the shown object is the closest to the robot's camera. In fact, a tracking routine  selects the pixels from the depth map that are closer to the robot, segmenting them from the background. However, this hypothesis limits the usage range to less than $1$m. The reason for this is twofold: firstly, considering relatively small objects, the blob of pixels depicting them from a distance that is further than $1$m is so small that the noise from the depth sensor might generate problems in the localisation of the closest pixels; secondly, with a greater distance between the robot and the teacher it is more difficult to satisfy the requirement of having the object as the closest element to the robot's camera. For these reasons, in this work we relax this assumption, by exploiting information about the human teacher. Specifically, the positions of the teacher body's key-points (given by considering the output of OpenPose jointly with that of the *Teacher recognition* module) are used to drive robot's attention toward the teacher. Precisely, we exploit the fact that the shown object's 3D points are close to the teacher's hand. Therefore, knowing the precise position of this latter allows to efficiently segment the held object using depth information.

# EXPERIMENTAL SETUP

With the aim of showing the effectiveness of the proposed approach with respect to the one presented in , we compare them in three different scenarios, with different levels of noise in the environment. In the rest of the paper, we refer our architecture as *Ours* whereas the one proposed in  as *Distance-based system*. The three different scenarios are:

1.  **Constrained**: it is the simplest scenario. Only the teacher is in the scene at less than $1$m far from the robot ($\sim0.5/0.8$m). Moreover, the object to learn is always the closest element to the robot's camera;

2.  **From afar**: this scenario represents the use case where the object is shown at a grater distance from the robot's camera ($\sim2$m). So doing, the object (which is still always the closest element to the robot) appears smaller and with a lower amount of detail.

3.  **With distractors**: in this last scenario, the teacher is $\sim 2$m far from the robot, as in the previous one, and is not alone in the scene. There are other objects and persons (called distractors) that can appear behind the camera or between the camera and the shown object. This scenario represents a more natural, less constrained, training setting.

For our experiments, we considered $9$ objects from the YCB dataset  (namely, *011_banana, 021_bleach_cleanser, 003_cracker_box, 035_power_drill, 025_mug, 006_mustard_bottle, 010_potted_meat_can, 037_scissors, 004_sugar_box*). To benchmark the system, we show each object to the robot, one at a time, in all the aforementioned scenarios, acquiring three different sets of image sequences (one for each scenario). Every set is composed of $9$ sequences (one for each considered object) having $300$ frames each (acquired at $\sim7$ frames per second). We use these sets of images to compare performance of *Ours* and *Distance-based system*. Firstly, we use them to evaluate the quality of the annotations provided by both approaches (Sec. ). Then, we use these annotations, together with the images, to train the *On-line object detection* module. We compare object detection performance of models trained with ground truth from *Ours* and the *Distance-based system* (Sec.  and ). To this aim, we test the models in a twofold way. Formerly, we show again the $9$ objects to iCub in the same conditions considered for the **Constrained** set (*iCub test set*) and we evaluate the quality of the object detection performance on $900$ frames ($100$ for each object) by comparing the predicted detections with a manual ground truth (note that, manual annotation of the sequences of the *iCub test set* was required to obtain ground-truth for benchmarking purposes, i.e. the ground-truth acquired in this way was not used for training). Lastly, as a further validation, we test the models trained with *Ours* and the *Distance-based system* on a subset of the Ho3D dataset [^6] . For doing that, we randomly selected one sequence for each of the $9$ objects (*ABF11*, *BB14*, *GPMF14*, *GSF14*, *MC1*, *MDF11*, *SiS1*, *SM2*, *SMu1*).

We report performance in terms of mAP (mean Average Precision) with the IoU (Intersection over Union) threshold set to $0.5$, as defined for COCO , using the publicly available implementation [^7].

# RESULTS

While the video submitted as supplementary material shows the functioning of the proposed application, in this section, we demonstrate its effectiveness reporting on the performed empirical evaluation.

## Annotations quality evaluation

Firstly, we aim at assessing the quality of the automatic annotations produced by the proposed approach. We do that by randomly sub-sampling $\sim270$ frames ($\sim30$ for each object) from each of the three sets of sequences described in the previous section (namely, *Constrained*, *From afar*, *With Distractors*) and we manually annotated them. We compare this manual ground truth with the automatic annotations produced respectively by *Ours* and the *Distance-based system*, to asses their quality (refer to Fig.  for annotation examples from both approaches). Tab.  reports the obtained results. As it can be noticed, in the **Constrained** condition, our approach outperforms the baseline of $\sim11\%$. Such a gap is mainly given by the different precision of the two systems for big objects, like for instance, the *003_cracker_box* and the *035_power_drill* (e.g., for the *003_cracker_box* our approach and the baseline achieve respectively an mAP of $92.6$% and $70.7$%). Indeed, depending on their pose with respect to the camera, those objects can be represented by a cluster of points in the image with significantly different distances from the depth sensor. In these cases, knowing the distance of the teacher's hand from the camera, is crucial to identify the image area to focus on and correctly segment all the pixels belonging to the held object. On the contrary, sharply thresholding on the depth level, without any other contextual knowledge (which is done in the *Distance-based system*) might lead to an imprecise segmentation. This might be addressed, to some extent, with an ad hoc hyper-parameters tuning of the *Distance-based system*, based on the size of the considered object and its distance from the camera. Nevertheless, a more general method that does not need any per-object adaptation, is preferable. Moreover, for the two more challenging conditions where the objects are shown from a greater distance and with distracting elements (namely, **From afar** and **With distractors**), the performance obtained by the proposed approach is significantly higher than the baseline. This shows the proposed method effectiveness in reducing the impact of disturbance elements during the data acquisition. Finally, these results demonstrate that the proposed method is able to automatically annotate the desired object in the image with a high mAP. Note that, even if this is not $100\%$, the obtained annotations can be used to train an object detection model. Indeed, it has been shown in previous work  that learning based object detection models are able to average out a certain level of noise in the dataset annotations. In the next sections, we evaluate performance of the entire proposed HRI-based learning pipeline.

|                    |      |      |      |
|:------------------:|:----:|:----:|:----:|
|        1-4         |      |      |      |
|      mAP (%)       |      |      |      |
|      mAP (%)       |      |      |      |
|      mAP (%)       |      |      |      |
| **Distance-based** | 73.0 | 51.2 | 19.3 |
|      **Ours**      | 84.4 | 70.5 | 68.0 |

Annotations quality evaluation. We compare annotations produced by Ours and Distance-based system.

## Object detection in a constrained scenario

In this section, we analyse the object detection performance of models trained with the proposed approach. We compare them with models trained with the *Distance-based system*, reported as reference baseline. The aim here is to verify the effectiveness of the proposed HRI-based learning pipeline when all the assumptions required by the baseline hold (i.e., close distance from the robot and no disturbance elements). To this aim in our experiments, we use the **Constrained** set of image sequences. We split the objects of the dataset by their size, identifying three different sizes: *Small* (*025_mug, 011_banana, 010_potted_meat_can*), *Medium* (*004_sugar_box, 006_mustard_bottle, 037_scissors*) and *Big* (*003_cracker_box, 021_bleach_cleanser, 035_power_drill*) and we train three different models one for each objects size. We do that to evaluate the impact of objects size in the automatic annotations collection and in the subsequent object detection training. We test the obtained model on the *iCub test set* (see Sec. ). Results are reported in Tab. . As it can be noted, for small and medium sized objects the models trained with the two different pipelines perform almost equivalently. This is due to the fact that the **Constrained** scenario satisfies all hypotheses required by the *Distance-based system* to function well (i.e., close distance from the robot and no disturbance elements). However, for bigger objects, the model trained with *Ours* performs significantly better. This is caused by the fact that, as explained in the previous section, the *Distance-based system* has issues in segmenting big objects due to the lack of contextual information. This annotation issue produces the object detection performance gap reported in the third column of Tab.  for big objects.

|                    |      |      |      |
|:------------------:|:----:|:----:|:----:|
|        1-4         |      |      |      |
|                    |      |      |      |
|      mAP (%)       |      |      |      |
|      mAP (%)       |      |      |      |
|      mAP (%)       |      |      |      |
| **Distance-based** | 73.2 | 63.4 | 58.5 |
|      **Ours**      | 72.4 | 65.4 | 80.7 |

Performance on the iCub test set, of models trained on the **Constrained** dataset. We compare models trained with Ours and Distance-based system.

```latex
\resizebox{8.4cm}{!}{
        \begin{tabular}{c|c|c|c|c|c|c}
            \cline{1-7}
                                                               & \multicolumn{3}{c|}{\textbf{\shortstack{From afar}}} & \multicolumn{3}{c}{\textbf{\shortstack{With distractors}}}  \\ \hline
              \textbf{\shortstack{Object size\\ { } }}                &  \textbf{\shortstack{Small\\ mAP (\%)}} & \textbf{\shortstack{Medium\\ mAP (\%)}} & \textbf{\shortstack{Big\\ mAP (\%)}}   &  \textbf{\shortstack{Small\\ mAP (\%)}} & \textbf{\shortstack{Medium\\ mAP (\%)}} & \textbf{\shortstack{Big\\ mAP (\%)}}  \\ \hline
              \textbf{\shortstack{Distance-based}}    &  43.0    & 55.3     & 24.0  &   29.0     &   37.0     & 32.0  \\ \hline
              \textbf{\shortstack{Ours}}              &  47.3    & 63.9     & 74.6  &   52.9     &    59.4    & 66.8 \\ \hline
        \end{tabular}
        }
```

## Object detection in a noisy environment

Finally, in this section, we remove the assumptions required by the *Distance-based system* and we prove that in these conditions, the proposed approach is critical to obtain good object detection performance. For doing this, we consider the **From afar** and **With distractors** sets of image sequences, we split them according to the objects sizes as described in the previous section and we use the resulting datasets for training object detection models. We test the obtained models on the *iCub test set* and, as a further validation, on a subset of the Ho3D dataset (Sec. ).

Results are reported in Tab.  for the *iCub test set* and in Tab.  for Ho3D. As it can be noticed, in general the models trained with the proposed approach perform better than the baseline on both test sets. Note that, in both cases the performance for small objects is the lowest. This is due to the fact that in the two test sets all the objects are depicted at a close distance, while in the two considered training sets they appear from a afar. This main difference especially affects performance for small objects because their appearance in the image is the most altered when taken from afar. Moreover, one may note that the average object detection performance in both scenarios is lower than the one obtained by using the **Constrained** training set. This is due to the fact that the *iCub test set* has been acquired in the same conditions used for the collection of **Constrained**, while the other two sets (**From afar** and **With distractors**) have been acquired in different conditions. This produces the so called domain shift  between the train and the test sets, resulting in the reported performance drop. Therefore, this is not caused by the annotations quality and it has been shown in previous work  that it can be solved by, for instance, integrating (i) a robot's autonomous exploration of the new domain and (ii) weakly-supervised learning techniques.

```latex
\resizebox{8.4cm}{!}{
        \begin{tabular}{c|c|c|c|c|c|c}
            \cline{1-7}
                                                & \multicolumn{3}{c|}{\textbf{\shortstack{From afar}}} & \multicolumn{3}{c}{\textbf{\shortstack{With distractors}}}  \\ \hline
               \textbf{\shortstack{Object size\\ { }}}               &  \textbf{\shortstack{Small\\ mAP (\%)}} & \textbf{\shortstack{Medium\\ mAP (\%)}} & \textbf{\shortstack{Big\\ mAP (\%)}}  &  \textbf{\shortstack{Small\\ mAP (\%)}} & \textbf{\shortstack{Medium\\ mAP (\%)}} & \textbf{\shortstack{Big\\ mAP (\%)}}  \\ \hline
              \textbf{\shortstack{Distance-based}}    & 30.8 & 58.3 & 25.0 & 26.3 & 47.0 & 25.0 \\ \hline
              \textbf{\shortstack{Ours}}              & 42.5 & 72.2 & 74.8 & 58.2 & 56.4 & 43.9 \\ \hline
        \end{tabular}
    }
```

In conclusion, the proposed approach proved to be effective for object detection training, being critical especially when the data acquisition conditions are not ideal. This allows for a more natural interaction with the teacher and to be more robust to distraction elements than the baseline.

## Object detection with multiple humans

For the sake of brevity we addressed the scenario with different humans in the video attached to the paper. We recorded the whole training pipeline having two people in the iCub's field of view proving that iCub is able to recognise the teacher and segment only the object in selected hand. Here we depict some sample frames in the Fig..

# CONCLUSIONS

In this paper, we propose to exploit social cues to improve current learning pipelines on robotic platforms. We present an architecture to train iCub to detect new objects in a teacher-learner setting, exploiting different human social cues. We focus on the social cues of the mutual gaze, gaze following and teacher recognition, overcoming several issues arose in previous work (mostly due to constrains in the interaction with the human). Specifically, we compare our approach with the learning pipeline in  reporting an improvement in performance in different environmental conditions. By removing several interaction constraints, our approach allows for a more natural human interaction with the robot. Therefore, our architecture can be used in less controlled setups, also in the experimental psychology, and might open new directions to study when a human feels engaged with a robotic partner. For example, in the setup of this work we can manipulate the delay in the actions/reactions of iCub to shed the light on the joint sense of agency.

# Acknowledgment

This work was supported in part by the Istituto Nazionale Assicurazione Infortuni sul Lavoro, under the project iHannes (PR19-PAS-P1).

[^1]: \*These authors contributed equally to the work

[^2]: $^{1}$Maria Lombardi, Elisa Maiettini and Lorenzo Natale are with Humanoids, Sensing and Perception Group, Istituito Italiano di Tecnologia, Genoa, Italy. `firstname.lastname@example.com, firstname.lastname@example.com, email@example.com`

[^3]: $^{2}$Vadim Tikhanoff is with iCubTech facility, Istituto Italiano di Tecnologia, Genoa, Italy. `email@example.com`

[^4]: <https://www.yarp.it>

[^5]: <http://vis-www.cs.umass.edu/lfw/>

[^6]: <https://github.com/shreyashampali/ho3d>

[^7]: <https://github.com/cocodataset/cocoapi/tree/master/PythonAPI/pycocotools>


abstract: We present the design of the simulation package Pluto, aimed at the study of hadronic interactions at SIS and FAIR energies. Its main mission is to offer a modular framework with an object-oriented structure, thereby making additions such as new particles, decays of resonances, new models up to modules for entire changes easily applicable. Overall consistency is ensured by a plugin- and distribution manager. Particular features are the support of a modular structure for physics process descriptions, and the possibility to access the particle stream for on-line modifications. Additional configuration and self-made classes can be attached by the user without re-compiling the package, which makes Pluto extremely configurable.
author: I. Fröhlich$^1$, T. Galatyuk$^1$, R. Holzmann$^2$, J. Markert$^{1,2}$,
B. Ramstein$^3$, P. Salabura$^{2,4}$ and J. Stroth$^{1,2}$
title: Design of the Pluto Event Generator[^1]

$^1$ Institut für Kernphysik, Goethe-Universität, 60438  Frankfurt, Germany
$^2$ GSI Helmholtzzentrum für Schwerionenforschung GmbH, 64291 Darmstadt, Germany
$^3$ Institut de Physique Nucléaire d'Orsay, CNRS/IN2P3, 91406 Orsay Cedex, France
$^4$ Smoluchowski Institute of Physics, Jagiellonian University of Cracow, 30-059 Kraków, Poland

# Introduction

Due to the fact, that experimental setups are usually not suited to cover the complete complete phase space, event generators are very important tools for experiments. Practically, the experimental physicist needs a tool in hand which allows to control almost all kinematic variables with a manageable user interface and to exchange the physics models which have to be compared to measured data.

In our contribution, we present the software structure of the Pluto event generator  originally developed for the HADES experiment  but successfully used by other collaborations in the hadronic physics field as well. Its recent redesign partly discussed in this contribution enhanced its flexibility and provide new features which allow to meet new challenges coming with the detector studies for the new FAIR experiments PANDA  and CBM .

Pluto is a collection of C++ classes, adding up to the framework of a simulation package for hadronic physics interactions in the energy regime up to a few GeV. It is launched interactively within the ROOT  environment, and makes use of ROOT only, without requiring additional packages. The focus is on streamlining particle generators by providing the tools to set up and manipulate particles, reaction channels, and complex reactions, as well as applying experimental filters on the reaction products, such as geometrical acceptance and kinematical conditions. Typical simulations may be executed with a few lines of input, with no expertise required on the part of the user. It standardizes simulations by providing a common platform that is accessible via the analysis environment (ROOT), with a straightforward user interface that does not inhibit non-experts in simulations.

The output may be analyzed on line, or further forwarded to a digitization package. The Pluto framework includes models for hadronic and electromagnetic decays, resonance spectral functions with mass-dependent widths, and anisotropic angular distributions for selected channels. A decay-manager interface enables "cocktail" calculations. An extensive particle data base is available, with capabilities to support user-defined ones. Various particle properties and decay modes are included in the data base. Thermal distributions are implemented, enabling multi-hadron decays of hot fireballs.

The general philosophy of Pluto is that on one hand the users should be able to drive the package in a very easy way, but on the other hand are allowed to customize it without much restrictions. This means that the steering application has the opportunity to add new decays/particles among with new models up to the incorporation of plugins[^2].

The structure and design of Pluto are sketched in Fig. . The framework can be roughly divided into 3 parts: a user interface for event production, a physics package, and an event loop interface. In this report we would like to concentrate on the latter two issues: the first one allows to add and/or select physics cases, where models can be either build-in models or runtime replacements, which will be discussed in Sec. . This area of Pluto works even outside the event loop for the calculation of parameters (mass distributions, decay widths,...). The other block allows for external particle decay and acceptance filter classes (see Sec. ) and filling of on-line histograms (see Sec. ).

As the interface for event production is described in detail elsewhere  in this context we will give only a short example. After all models have been configured accordingly, an elementary reaction chain is defined with a character string containing the reaction products. To perform, e.g., the $\Delta^+$ production in the $pp$ reaction with a consecutive Dalitz decay as discussed in the following sections, the commands as defined below are sufficient:

     PReaction *r = new PReaction("1.25","p","p",
                                  "p D+ [p dilepton [e+ e-] ]");
     r->Print();
     r->Loop(10000);

with the number 1.25 as the kinetic beam energy in GeV, the beam and target particle indicated by following two options and finally a string describing the decay. Particles are defined via their data base name (as "`p`" for a proton or "`D+`" for a $\Delta^+$). The setup of the reaction channels and the attachment of all selected distributions and models is done by the framework, with the possibility to dump the complete chain and its models via the `Print()` method. Finally, the command `Loop(n)` produces $n$ events.

Then, event-event-by-event, the individual steps in each reaction are executed via a common interface of all classes making distributions (see  for details) which do the calculation of final particle tracks (also the unstable ones) which are stored in an array.

# The models

## Architecture

In the Pluto framework, each physics process is described by a number of objects, instantiated from dedicated classes. Beside the objects which communicate with the event loop and set the 4-momenta of the resulting particle tracks (to form the above-mentioned "distributions"), Pluto offers methods to make calculations including various channels and their accompanied functions in advance to (or event without) the event-loop. For this purpose, a relative data base has been included, which connects particles and decays to "models". At least one (so-called "*primary*") model is needed per decay in order to describe the respective mass shape (usually calculated by a mass-dependent Breit-Wigner in a recursive way) and the partial/total decay width. Each selected model can be accessed from all locations of the software package as well as by the user.

The choice which of the allover offered physics models are linked to the data base can be done inside the production ROOT-macro by calling a distribution manager (for details about the handling see ), which itself takes care that only one primary model per decay and particle is selected, respectively. Moreover, Pluto offers the user the freedom to attach user-defined models based on its own sources or even new classes (written in C++) at runtime, thereby replacing built-in models. These features make Pluto an extremely versatile and highly configurable tool for simulations, while guaranteeing overall consistency at all times.

One specific feature of Pluto is that we do not use monolithic decay models only but allow for the splitting of the underlying physics process into different models in a very granular way (e.g., to exchange form factors or total cross sections). This turned out to be a very important tool in order to check various scenarios along with measured data. Therefore Pluto allows for the attachment of secondary models for all kinds of purposes. Here, a secondary model is an object for a particle/decay returning a (complex) number as a function of a defined number of values. We do not attempt to have a base class for each kind of function and model (e.g., a total cross section base class, a base class for $d\Gamma / dm$, etc.) because this makes further extensions difficult. Our approach is as follows: for each class of secondary models a new entry (defined by a unique name) is added dynamically to the data base. Consequently, each model is bound to two data base keys: one to link it to a decay (or particle) and one more for the type of secondary model.

Anyhow, there the user does not need to take care about these issues, as the following template example shows.

## Template example

In Pluto, each model is inherited from the base class `PChannelModel` with a lot of useful features. In order to offer the result of the calculation the `GetWeight()` method has to implemented. Let us create such an example class as a template for the following discussion and future developments:

     class PHelperFunctionTest : public PChannelModel {
     public:
       PHelperFunctionTest(const Char_t *id, const Char_t *de,
                           Int_t key);
       Double_t GetWeight(Double_t *x, Int_t *opt=NULL);
     }

with an realization of the (in this case 1-dimensional) function:

     Double_t PHelperFunctionTest::GetWeight(Double_t *x,
                                             Int_t *opt) {
       Double_t xx = x[0];
       val = ...
       return val;
     };

Objects can be instantiated to work as a model for specific physics processes by the usage of a parser residing inside the constructor of the base class:

     PHelperFunctionTest *obj = new
       PHelperFunctionTest("unique_name@D+_to_p_dilepton/helper",
                           "Test",-1);

Here, the ` unique_name` is used to identify the object inside the distribution manager. The string hereafter binds the new object to a decay which is in this case $\Delta^+ \to p \gamma^*$. The optional appendix `/helper` enables the new object to be a secondary model of this type.

Let us now assume, the primary model has to access its helper function. In this case it is sufficient to execute the following command at a single place (an `Init()` function is available for that):

     PChannelModel *sec_model=GetSecondaryModel("helper");

and execute the daughter method `sec_model->GetWeight(value)` whenever required. The access to alien secondary models is possible as well but a further discussion on this topic would exceed the scope of this report.

## Example: form factors

The power of such a splitting can be demonstrated best with an real example. For this purpose let us use the study of the electromagnetic $\Delta$ Dalitz decay $\Delta \to N \gamma^* \to N e^+ e^-$. As explained in the introduction Pluto reads this as a multi-step process: in the first step, the $\Delta$ mass is sampled using a mass-dependent width. In the second step, the already sampled $\Delta$ mass $m_{\Delta}$ is used by the following Dalitz decay model to sample the di-electron (=virtual photon) mass $m_{ee}$.

By using a dedicated plugin:

     makeDistributionManager()->Exec("dalitz_mod: krivoruchenko");

the following model for the di-electron mass spectrum is enabled, using the prescription of Ref.  with:

$$\label{eqn:wolf}
\frac{d\Gamma^{\rm \Delta\to Ne^+e^-}_{m_\Delta}(m_{ee})}{dm_{ee}}
=\frac{2\alpha}{3\pi m_{ee}}
\Gamma^{\Delta \to N \gamma^*}_{m_\Delta}(m_{ee})$$ and the decay rate given by Krivoruchenko *et al* : $$\begin{aligned}
\label{eqn:delta_dalitz_decay_kriv}
  \nonumber \Gamma^{\Delta \to N\gamma^*}_{m_\Delta}(m_{\gamma^*})
  & = &  \left( G^{\Delta \to N\gamma^*}_{m_\Delta}(m_{\gamma^*}) \right)^2 \\
  & & \nonumber \times \frac{\alpha}{16}
\frac{(m_\Delta + m_N)^2}{m_\Delta^3m_N^2}
\sqrt{y_+ y_-^3}, \\ y_\pm & =& (m_\Delta \pm m_N)^2-m_{ee}^2
\end{aligned}$$ where the index $N$ refers to the produced nucleon, $e$ is the electron charge, and $G^{\Delta \to N\gamma^*}_{m_\Delta}(m_{\gamma^*})$ represents the electromagnetic transition amplitudes. Here, from the software point of view, it is important to know that this factor is a separated object in Pluto making the addition of new complementary descriptions very easy. This avoids the usage of nasty flags which is error-prone. E.g., following the QED approach (assuming that no hadronic transition form factor is involved) from Ref. gives:

$$\label{eqn:form_factor}
  \left(G^{\Delta \to N\gamma^*}_{m_\Delta}(m_{\gamma^*})\right)^2
  = G_M^2 + 3G_E^2 + \frac{m^2_{\gamma^*}}{2m_\Delta^2} G_C^2$$ with $G_M$, $G_E$, and $G_C$ as the magnetic, electric and Coulomb transition form factors.

Alternatively, one can switch to the VMD-like form factor of Iachello and Wan  which is already implemented and will be discussed in detail in a further, separate publication.

After activating this plugin, both above-mentioned versions (VMD and QED) are offered, appearing as objects with their unique names in the directory-like distribution manager structure.

    makeDistributionManager()->Print("vmd");
    --------------------
    PDistributionManager
    --------------------
      root                Root group: 494 objects (of 504),
                            subgroups: 9
         particle_models  Mass sampling of particles : 27 objects
                            (of 29)
         decay_models     Phase space mass sampling & decay partial
                            widths: 219 objects (of 225)
         polar_angles     Polar angles in elementary particle
                            production: 10 objects (of 10)
         genbod_models    Momentum sampling: 220 objects (of 220)
             ...
         vmd              VMD form factors: 0 objects (of 2)
    ( )  D+_vmd_ff        VMD ff for D+ -> p e+e-
     (X) D+_qed_ff        QED ff for D+ -> p e+e-
             ...
         qed              QED form factors: 2 objects (of 2)

Possible conflicts between the QED and VMD models are indicated. The user can now activate all models inside the directory via:

     makeDistributionManager()->Enable("vmd");

# The event loop interface

The aim of the event loop interface is to allow access to the particle bulk before it is written to disk by providing an interface for bulk decays (also using 3$^{rd}$ party event generators such as Pythia ), file access, implantation of embedded particles for detector studies, and filter classes. It works - like indicated in Fig.  - by stacking individual bulk classes (by `PReaction::AddBulk(PBulkInterface * obj)`) and making use of the base-class `Modify()` method which handles the embedding and removing of particles or mark them as decayed/invalid.

Like in the previous section, user classes can be compiled - if needed - and added on-the-fly, which could be very helpful in the context of FAIR experiments studies, as the following applications show:

## File access:

Particle tracks might be added from (or written to) user files in any self-defined format. This feature makes Pluto open for other collaborations and enables it to be used as an "afterburner" for externally created simulation data or output of transport models.

## Embedded particles:

Test particles can be embedded into a background reaction which is an important tool for detectors studies, e.g. to test tracking in a high multiplicity environment. Moreover, among with the previous feature the mixing of embedded particles with either real events or 3$^{rd}$ party simulation is one of the applications. In the first case, the reconstructed vertex of the analyzed experimental data is used to simulate rare probes (as the di-electrons) at exactly the same vertex. In this way the efficiency of these probes can be calculated with the best background one can have, namely measured data.

## Bulk decay:

In addition to a well defined reaction chain, Pluto offers free "bulk" decays, i.e. the decay of (embedded) particles following the calculated mass-dependent branching ratios. At this stage it is possible as well to employ 3$^{rd}$ party event generators. Only to give an example: Pluto can generate embedded particles like resonances with a selected distribution and the Pythia package (or Pluto itself) can be used for their realistic free decay.

## Detector acceptance class:

The particles created in the previous steps can be pre-filtered using a "detector acceptance" user class following a class definition as outlined below:

     class PMyDetector : public PBulkInterface {
     private:
       Double_t *acc;
     public:
       Bool_t Modify(PParticle ** stack, int *decay_done, int * num,
                     int stacksize);
     };

     PMyDetector::PMyDetector() {
       acc = makeStaticData()->GetBatchValue("acc_value");
     };

with `acc_value` as a booked batch value (see below for more information) and, e.g.:

     Bool_t PMyDetector::Modify(PParticle ** stack, ... ) {
     *acc=1.;
     for (int i=0; i< *num; i++) {
       PParticle * cur = stack[i];
       if ("check for cur") *acc=0.;
       }
     }

# Projecting events

## General idea

The usual way to analyze generated events (beside a full Monte-Carlo production including digitization) is to open the simulation file with an appropriate analysis macro, loop over the number of events, read the particle objects and do the required operations inside the loop. E.g., in hadronic physics, in particular for exclusive studies, usually missing and invariant masses are reconstructed, and possibly after boost operations angular correlations are investigated. In addition, signal selection has to be done not only on a single track level but using particle correlations (such as a invariant/missing mass selection). This prevents to use the features provided by `TTree::Draw()` only, as these are fixed to single track properties.

The main aim of the "`PProjector`" (see Fig. ) is to offer a simplified analysis method following a similar idea as `TTree::Draw()` in order to project particles (and their correlations) onto on-line histograms and/or `TNtuple`s by making use of the bulk interface described in the previous section. This means, no analysis macro is needed for a large number of typical observables.

There are two ways to include the projector into the event loop. Either one or more of its instances are added to the bulk interface (for an alternating mixing with other classes), or, which is sufficient for the most cases, the short version `PReaction::Do(char * command)` can be used, where `command` follows a C++-like (but more simple) syntax. Each of these commands - just forming steps in a batch - is executed inside the event loop.

## Histograms

To fill histograms one can use the extended method `PReaction::Do(THx * histo, char * command)` where `histo` points to a 1/2/3-dimensional histogram.

In this case the command has to define the key variables `_x, _y` and `_x` as the respective axis value, depending on the chosen dimension of `histo`. The Pluto particle array objects can be accessed via its Pluto particle name inside brackets as `[name,num]` with the optional `num` as the consecutive number of equal particles inside each event. All objects can be combined with a number of build-in methods, moreover all browsable methods of the ROOT-class `TLorentzVector` can be employed, as the following example indicates:

     r->Do(histo1,"_x = [D+]->M()");

which fills the 1-dimensional `histo1` with the $\Delta^+$ mass. Invariant masses (composite particles) can be formed by adding the objects:

     "_x = ([e+] + [e-])->M2()"

and stored for a consequent use in semi-colon separated sub-commands without requiring a "new" operator:

     "dilepton = ([e+] + [e-]); _x = dilepton->M2()"

Let us now come to a more complex example, which is the calculation of the polar emission angle of one of the outgoing protons in a $p+p$ reaction (N.B. Pluto treats the c.m. system as a particle named `p + p`):

     "p1 = [p,1]; p1->Boost([p + p]); _x = cos(p1->Theta())"

In this example, in the first sub-command the first appearing proton is copied to the object `p1` which is furthermore boosted into the c.m. energy frame and in the last step the cosine of its polar angle is written as a value to the x-axis.

## TNtuple in- and output

The projection to `TNtuple`s turned out to be very useful as it allows to make additional selections (and definition of the histogram axis) afterwards which is a very simple task using the `PProjector` following the syntax as described above. Let us now assume, the momenta of simulated $\eta$'s should be stored. Then the first step would be to define the `TNtuple` with branch names:

     TNtuple *ntuple =
       new TNtuple("ntuple","...","eta_px:eta_py:eta_pz");

Hereafter, the `Do()`-method has to be called like in the above-indicated example but with a pointer to a `TNtuple`-object instead of a histogram and the variable names of the ntuple instead of the axis names:

     "eta_px = [eta]->Px(); eta_py = [eta]->Py();
        eta_pz = [eta]->Pz()"

The filling of the ntuple is done by Pluto, only (but on purpose) the opening and closing of the root-file has to be called externally.

In addition to the writing of `TNtuple`s it is also possible to read (back) any ntuple, modify its values and write it back or make histograms. Let us use the ntuple of the previous example which uses the ntuple (usually from a file) and defines an *empty* reaction, but attaches the ntuple as a 3rd party input:

     PReaction my_reaction;
     my_reaction.Input(ntuple);

The next step is to use the branches of the ntuple in the batch command line to, e.g., reconstruct the 4-momentum:

     "myeta = P3M(eta_px,eta_py,eta_pz,0.54745)"

with `P3M` as the short syntax for a vector constructor with 3 momenta and the particle mass as parameters. At this point any kind of mathematical operation available via `TFormula` can be applied as well. Finally using `Loop()` without a number reads all events from the ntuple.

## Filter commands

The employment of batch commands allows to define on-line filters on single particle properties as well as kinematic variables which are accessible as indicated above. These filters can be used to select events for single histograms only or forwarded to the event loop to apply physical constraints, before a full event is written to disk, hence reducing the event file. Variables prepared in a detector class as outlined in Sec.  can be integrated very easily.

This is realized by using the `if`-command, followed by the variable or a combined condition based on particle observables. The remaining batch sub-commands are executed only under a positively evaluated condition. It goes without saying that the condition statement can combine all methods among with the established `TFormula` syntax:

     "if(obj->P() > 0.3 && obj->P() < 0.6); dowhatever..."

In this case, histograms and ntuples in the `Do()` method are only filled under the defined condition, or, "dowhatever" can be used to re-define variables.

In this context, it should be noted, that variables, starting with "`#`" are interpreted by the Pluto event loop as a condition for the final event file. In our next case, an event is accepted if both the electron and the positron are produced with polar angles in the range of 18 to 85 degrees:

     "theta_ep = ([e+]->Theta() * 180.)/TMath::Pi()"
     "theta_em = ([e-]->Theta() * 180.)/TMath::Pi()"
     "#filter = 1; if(theta_ep<18 || theta_ep>85 || theta_em<18
                      theta_em>85); #filter = 0"

by this means it is only a single step to activate the detector class `MyDetector` from Sec. :

     "#acc_filter = acc_value"

and optionally combine the prepared acceptance flag with our kinematical conditions.

# Summary

In summary, we presented new developments for the Pluto event generator which moves this package from being a pure event generator to a comprehensive framework. The particular features are the bookkeeping of various models and the user interface for their simple selection. The new event loop interface allows to mix several sources, to define and enable online selections, project to histograms and to make a lot of operations. Detector acceptance and resolution can be included without much overhead, which is of particular importance for detector studies in the context of the FAIR project.

In the near future, we will implement general purpose resolution scenarios using a common data format and file readers to transport models (like UrQMD  and HSD ). This will allow to test very rapidly the performance of different detector configuration options.

# Acknowledgements

This work was supported by the Hessian LOEWE initiative through the Helmholtz International Center for FAIR (HIC for FAIR). We would like to thank B. Kämpfer for reading and commenting the manuscript.

[^1]: Presented at the 17th International Conference on Computing in High Energy and Nuclear Physics

[^2]: The PC gaming industry, making benefit from charge-free down-loadable packages, calls such extension features also "mod".


abstract: Dynamic bipedal walking on discrete terrain, like stepping stones, is a challenging problem requiring feedback controllers to enforce safety-critical constraints. To enforce such constraints in real-world experiments, fast and accurate perception for foothold detection and estimation is needed. In this work, a deep visual perception model is designed to accurately estimate step length of the next step, which serves as input to the feedback controller to enable vision-in-the-loop dynamic walking on discrete terrain. In particular, a custom convolutional neural network architecture is designed and trained to predict step length to the next foothold using a sampled image preview of the upcoming terrain at foot impact. The visual input is offered only at the beginning of each step and is shown to be sufficient for the job of dynamically stepping onto discrete footholds. Through extensive numerical studies, we show that the robot is able to successfully autonomously walk for over $100$ steps without failure on a discrete terrain with footholds randomly positioned within a step length range of $[45 : 85]$ centimeters.
author: Avinash Siravuru, Allan Wang, Quan Nguyen, and Koushil Sreenath [^1] [^2] [^3]
bibliography: newrefs.bib; ref.bib; Quan_ref.bib
title: Deep Visual Perception for Dynamic Walking on Discrete Terrain

# Introduction

## Problem Definition

The objective of this work is to build and systematically evaluate a Deep Visual Perception system to be used by a planar dynamic walking robot in order to autonomously walk on discrete terrain. We build a realistic visual simulator to generate the robot's first person view and combine it with an accurate physics simulator of the bipedal robot walking on discrete terrain. The physics simulator also contains an inner-loop safety-critical controller that can generate stable and safe limit cycle walking of a desired step length . In this setup, we train a deep neural network to estimate the step length (distance to the next stepping location) using a single sampled image preview that is obtained at the beginning of each step. Detecting footholds and estimating distance is a classic object localization problem similar to object grasping in robotic manipulation, however in the case of locomotion there are additional challenges due to the time-critical and safety-critical nature of the problem.

Note that, we limit our attention to only autonomous planar walking, and accordingly, only predict step length information. This simplification allows us to keep the focus on the visual simulator development, custom CNN design (to bound worst-case estimate) and perception-control integration. However, the method itself can be extended to 3D walking without loss of generality as evidenced by (a) our prior work on dynamic walking over terrain with varying step width and step height in addition to step length , and (b) the perception system in this paper that takes an image rendered from a 3D scene as input without making any geometric simplifications due to the planar walking.

## Motivation

Conventionally, robot perception involves parsing the entire scene, labeling objects of interest, and feeding this information for planning and control. As the number of tasks increase and the decision-making time drops, searching the entire scene for all tasks is expensive and trade-offs are inevitable. Reactive vision-in-the-loop control for sub-tasks, wherever possible, reduces overhead on higher-level planners and injects more dynamism into the system. The objective of this work is to serve this need for the sub-task of walking safely on discrete terrain.

In fact, experiments with humans and cats have shown that during over $50-60 \%$ time of the gait cycle, gaze is invested in target detection and the higher-level spatial navigation problem . When required to walk on complex terrain requiring accurate foot placement, humans operate with intermittent visual samples of the foothold location and use the information in a feed-forward manner to adjust step length just $1-2$ steps a priori . For walking on discrete terrain, the key finding here is that, instead of active modulation of gait through visual feedback during the entire stance phase, humans prefer to adjust their gait one-two step ahead using an intermittent visual preview and execute an energy-efficient ballistic motion . If the foothold remains constant, continuous visual feedback may even be unnecessary. Our work is inspired by these biological findings and we wish to elicit similar behavior from the dynamic bipedal robot ATRIAS. Our paper makes the following contributions:

1.  We present a custom deep convolutional neural network (CNN) architecture for the task of estimating next step location given a sampled visual preview of the terrain at the end of the current step. A synthetic outdoor dataset with dynamic real world textures, varying lighting conditions and camera positions is also developed. The network, trained on this dataset, returns an average prediction error of $1.62$ cm and a worst-case error of $10.38$ cm.

2.  We integrate the CNN-based step length estimator with an inner-loop safety-critical controller to enable vision-in-the-loop dynamic walking on discrete terrain. The robot with the visual step-length estimator and the safety-critical controller is shown to successfully walk at least 100 steps without failure, with the step lengths randomly sampled from a uniform distribution of $45-85$ cm.

# Related Work

**Deep Learning for Robotic Vision:** Recent advances of Deep Learning in fields like computer vision, natural language processing, speech recognition, etc. has lead to tremendous research interest in extending these gains to robotics. Large amounts of data available for network training and parallel computation helped accelerate this effort. From learning end-to-end visuomotor policies for object manipulation , to learning to fly UAVs in cluttered environments , or in self-driving cars , deep learning is impacting all major robotics sub-domains. In humanoid robotics also, CNNs were used for innovative applications like surface friction estimation from images, to help in slip prediction . However, it is very challenging to build end-to-end fully data-driven policies for stable and safe limit cycle walking, let alone on discrete terrain.

**Perception in Legged Locomotion:** Perception for bipedal locomotion primarily focused on foot-step planning for statically stable or linear dynamical model-based walkers. Usually, LIDAR-camera combination is preferred in this case. Accurate high resolution depth data obtained from Lidar is used for safe footstep detection and planning . Unlike these walkers, dynamics robots have point feet, move much faster and therefore need faster execution and the ability to pick footholds of any size. This makes the search problem harder on the full 3D map. Vision-in-the-loop walking with gait adjustment (comparable to our approach) was implemented on a Quadruped in to operate on steep slopes and dense vegetation. However, the problem of discrete terrain is not addressed. We believe our solution is complimentary to their effort and a combined solution could pave the way for true rough terrain navigation.

# Robot Model and Controller Summary

Having presented an overview of related work, we will now briefly develop the dynamical model and controller for achieving stable walking with precise foot placements.

## Dynamical Model for Walking

We consider the ATRIAS bipedal robot with configuration $q$ and state $x = (q,\dot q)$ that denotes the generalized positions and velocities of the robot, with $u$ denoting the joint torques. A hybrid model of walking can then be expressed as $$\begin{aligned}
\label{eq:hybridsystem}
\left\{\begin{array}{llll}\dot{x} & =& f(x)+g(x)u  & x \notin {\cal S}
\\ x^+ &=& \Delta(x^-) & x \in {\cal S}, \end{array}\right.
\end{aligned}$$ where ${\cal S}$ is the switching guard, $\Delta$ is the reset or impact map, and the superscripts $-$ and $+$ denoting pre- and post-impact variables respectively. A detailed description of the robot and a derivation of its model can be found in .

## Periodic Gait Design using Virtual Constraints

Virtual constraints are kinematic relations that synchronize the evolution of the robot's coordinates via continuous-time feedback control, thereby simplifying control of high degree-of-freedom systems, . Virtual constraints are expressed as an output vector $$\begin{aligned}
    \label{eq:yPhase}
    y(q) = h_0(q) - h_d(s(q),\alpha),
\end{aligned}$$ to be asymptotically zeroed by a feedback controller, with one virtual constraint typically imposed per each actuator. Here $h_0(q)$ specifies the variables to be controlled, and $h_d(s,\alpha)$ specifies the desired evolution of the controlled variables, parametrized by the the coefficient $\alpha$ and the gait phasing variable $s$ which goes from zero at the start of the gait to one at the end of the gait.

A nonlinear constrained optimization is used to find the coefficient $\alpha$ so as to create a periodic orbit satisfying a desired step length, while respecting physical constraints on torque, motor velocity, and friction cone. The cost function is taken as the integral of squared torques normalized by step length, $$\begin{aligned}
    \label{eq:cost}
    J= \frac{1}{L_{step}}\int_{0}^{T} ||u(t)||^2_2~ dt,
%   J= \int_{0}^{T} ||u(t)||^2_2~ dt.
\end{aligned}$$ and the constraints for the optimization are formulated as given in Table , see for more details. The optimization is solved through a fast direct collocation framework from .

|                                |                        |
|:------------------------------:|:-----------------------|
|          Motor Toque           | $|u|\le 7$ Nm          |
|         Impact Impulse         | $F_e\le 15$ Ns         |
|         Friction Cone          | $\mu\le0.6$            |
| Vertical Ground Reaction Force | $F_{st}^v\ge200$ N     |
| Mid-step Swing Foot Clearance  | $h_f|_{s=0.5}\ge0.1$ m |

Optimization constraints.

Since walking over stepping stones involves changes in the step length, one way to easily transition between different step lengths is to create a 2-step periodic gait that explicitly considers the step length of the current step as well as the subsequent step. The optimization process presented above can be used to design a 2-step periodic orbit with step lengths ($l_0$, $l_1$) with coefficient $\alpha(l_0, l_1)$ as shown in Fig. .

With a collection of 2-step periodic orbits, We can then transition at each step between multiple 2-step periodic orbits in a MPC-like manner with small transients. However, to prevent an explosion of the number of periodic gaits that need to be optimized for, here we use only four 2-step periodic gaits corresponding to step lengths $(l_{min},l_{min}), (l_{min},l_{max}), (l_{max},l_{min}), (l_{max},l_{max})$ and use bilinear interpolation to find the coefficients $\alpha$ for a desired gait of step lengths $(l_0^d, l_1^d)$ as shown in Fig. . This work builds off recent work on periodic walking gait libraries in .

## Feedback Control

The presented optimization results in a desired walking gait encoded through $h_d(s(q),\alpha)$ in . The goal for the feedback control is to drive $y(q) \to 0$. In this work, we use a nonlinear feedback controller to enforce exponential stability for the hybrid system through input-output linearization, . If $y(q)$ has vector relative degree 2, then its second derivative can be expressed through Lie derivatives as, $$\label{yddot}
\ddot{y} = L^{2}_{f}y(\q,\dq) + L_{g}L_{f} y(\q,\dq) ~ u.$$ We can then apply the following pre-control law $$\begin{aligned}
\label{expControl}
u(q,\dot{q}) = u^{*}(q,\dot{q}) + (L_{g}L_{f} y(q,\dot{q}))^{-1} ~ \mu,
\end{aligned}$$ where the nominal feedforward component $u^{*}(q,\dot{q}) := -(L_{g}L_{f} y(q,\dot{q}))^{-1} L^2_{f} y(q,\dot{q})$ with feedback $\mu=-K_p y-K_d \dot y$ stabilizes the system. This combination of the 2-step periodic gait libary and the feedback controller results in precise footplacements with specified step lengths in a safety-critical manner. Formal guarantees on safety for dynamic walking on discrete terrain can also be achieved through control barrier functions .

# Direct Perception for ATRIAS

Having presented the robot's dynamical model and the safety-critical controller designed to walk on discrete terrain when provided with accurate step length information, we will now present a systematic way to build and train a deep visual perception model that estimates the step length from a single monocular image. The system will take an input of the upcoming terrain through a front-facing camera at the beginning of each step. This image is to be fed to a CNN to estimate the step length for the next step. The step length estimate is then fed to the gait-library based controller to enable the robot to precisely land on the next foothold.

This CNN-based deep perception model has two critical components. Firstly, we need a large corpus of step-length annotated imagery of the robot's front person view while walking. This dataset is used to train the model for accurate step length estimation. The systematic methodology used to create this synthetic dataset is described in the next subsection. Secondly, we need a suitable deep neural network architecture that can best approximate the complex non-linear mapping from image to step length estimate. The network needs to be tuned methodically to not only obtain the best test accuracy but also bound the worst-case prediction. These details are summarized after the next subsection.

## Dataset Generation

To generate the image dataset, we use a popular open-source graphics software called Blender to programatically generate realistic scenes. To create a discrete terrain scene, we need four key details: 1) Camera location and intrinsic parameters 2) Stepping stone location, 3) Lighting model and location, and 4) Color and texture information of both the stepping stone and background. These parameters will be randomized in ranges larger than what the robot may encounter in order to account for error accumulation over time.

The above four parameters are randomized for image generation in the following manner:

**Camera Location**: The camera location is measured with respect to the stance foot position. For each image, we randomly sampled from a range of $[-10:20]\times[-10:10]\times[80:120]$ (cm) to obtain the x, y, z offsets of the camera from the stance foot, respectively. The ZED Stereo Camera model is used for rendering the images. However, since we only focus on the step length, we only generate monocular images for this study.

**Stone Location**: From , we note that, the robot was able to walk on a discrete terrain where the step lengths ranged from $[20:90]$ (cm). True step length is the distance from the robot's stance foot to the next stone's center. For this work we uniformly sample step lengths from $[15:95]$ (cm) range and arrange the stones in a single column accordingly. Based on the width of the robot, we spread out stones $[30:35]$ (cm) away of the robot's center-line. Moreover, they are alternately positioned on either sides of the center-line, as shown in Fig. . Finally, the stone size itself is varied randomly between $[10:20]$ (cm) in length and $[60:90]$ (cm) in width, respectively. However, the stone height is kept fixed at $15$ cm.

**Lighting Properties:** The light source is always facing the current stepping stone. However, its position is randomly chosen from a semi-spherical dome-like space above the robot with a $6$ meters constant radius to the current stone-center.

**Texture:** Here we select real world textures for both stones and background. We chose $12$ unique textures comprising of grass, sand, water, pebbled terrain, etc. for the background and $14$ unique textures for the stone including granite, cement, brick, wood, etc. Texture samples are shown in Fig. Each scene is rendered by randomly sampling a texture pair from the available collection.

The images are rendered with a resolution of $223 \times 223$ pixels. Additionally, we crop the left and right $15 \%$ of the image to further reduce computational overhead. The final image resolution is $223 \times 149$. We generate $40,000$ images and call it the Synthetic Outdoor Dataset (SOD). Sample images are shown in Fig. . Having presented the dataset generation details, we will present the neural network architecture design next.

## Network Architecture Summary

For training our object detector, we propose a custom neural network architecture. It consists of six convolutional layers of $32$ filters each, followed by two dense layers, both with $256$ neurons each. Unlike traditional designs where the number of features maps increase with depth, we found that, a constant number of feature maps throughout does better and needs fewer parameters. The kernel size of each filter is $(4\times 4)$. Batch Normalization and Max Pooling are applied after every two convolutional layers. Additionally, we apply Batch Normalization just before the final output layer as well. We use *relu* activations in all the layers except the last one, where we use a *linear* activation function instead. Fig.  summarizes the CNN architecture. The detector is trained using the *Mean Squared Error* loss and the *Adam* optimizer. We use a learning rate of $1e-4$ along with a suitable learning rate decay policy. We use Keras API with TensorFlow backend to build and train our model. We trained for $40$ epochs with a batch size of $50$, on a Intel i7 machine with an NVIDIA Titan X GPU. The model has roughly $2.5$ million trainable parameters. Finally we split the dataset into Training, Validation and Test sets, each comprising of $28900, 5100, 6000$ images, respectively.

The step length is estimated as follows: Using the joint encoders on the robot, the location of the camera is first estimated. The deep neural network only learns to predict the distance from the camera to the next stone center. Therefore, the predicted step length is the sum of these two distances. Note that the camera position information is not used during training.

While deep networks have remarkable function approximation abilities, they have many tunable hyper-parameters whose fine-tuning critically impacts network performance and generalization ability. In the next section, we systematically outline our network design and customization process while examining the impact of each hyper-parameter on reducing worst-case test error.

## Hyper-parameter Search

Deep neural networks have a very high dimensional hyper-parameter space, where almost every single building block can be optimized. Most applied deep learning papers use existing architectures and leverage their transfer learning properties. Few papers explain in sufficient detail, the impact of each hyper-parameter on the learning outcomes. Unfortunately, hyper-parameter choices don't always generalize to all problems and it is therefore worthwhile to carefully tweak and specifically examine them for individual problems. Important insights drawn from this exploration for our problem are summarized below. Note that, all the results reported below are on the test set.

-   Roughly $89\%$ drop in error occurs within the first $20$ epochs. Therefore, a wider hyper-parameter coarse search was carried out on models trained for $20$ epochs while for the finer search, the models were trained for $40$ epochs.

-   As already identified in , Dropout with any probability or placement in the network worsens performance.

-   Batch Normalization really boosts performance and results in around $33\%$ drop in mean absolute error. More interestingly it leads to over $55\%$ drop in the worst-case prediction error (or the maximum prediction error on the test set).

-   Using L2 Regularization (default is $0.01$) for only the fully connected layers helps further reduce the worst-case prediction error.

-   We tested the Architecture with 5 kernel sizes, $(2,2), (4,4), (6,6), (3,3), (2,4)$. We observed that rectangular kernels had the largest worst-case error, followed by the $(3,3)$ kernel. Surprisingly, even numbered kernels did a better job, against conventional wisdom. The best kernel was $(4,4)$.

-   Adding an extra convolutional block (ie., two additional convolutional layers followed by a max pool and a batch norm) increased the mean absolute error. Unlike classification tasks where depth almost always helps, in regression tasks, localization accuracy is affected by max-pooling layers beyond some depth. Therefore, for regression tasks one must find the sweet spot between depth (complexity) and accuracy.

Finally, based on the above mentioned hyper-parameter search, an optimal network architecture is designed and it is trained with the synthetic outdoor dataset. The qualitative and quantitative results obtained are presented in the next section.

# Results and Discussion

In this section, we study the performance of the deep perception model and then integrate it with a physics-based simulator and gait-library based controller to numerically realize and analyze autonomous dynamic walking on randomly generated discrete terrain.

## Step Length Prediction Performance

Once trained, we expect the step length predictor to accurately detect the next stepping stone and output it's distance in centimeters. Note that, each image will have anywhere between $1-5$ stepping stones. Therefore, even though they have the same texture and geometry, our perception framework needs to overlook other stones and actively seek out the first one. In this situation, we believe perspective distortion helps in better distinguishing the stone of interest. Further, due to the safety-critical nature of this problem, in addition to describing network performance based on mean squared error, we will also report the variance and the worst-case prediction.

Error is unavoidable in function approximation. However, in a safety-critical scenario like discrete terrain walking, the default approach of choosing the network that gives the least mean squared error could be detrimental as the worst-case estimate could still be off the safety limits. In order to avoid this issue, in this work, hyper-parameters were tuned with the objective to find the least possible worst-case estimate with the available dataset. The performance of the CNN was evaluated on the $6000$ sample test data and is visualized in Fig.  and summarized in Table .

```latex
\resizebox{\columnwidth}{!}{
\centering
\begin{tabular}{cccc}
\toprule
 Test Avg. Loss & Std. Dev. of Loss  & \% Above 5 cm & Max Pred. Error \\
 \quad (in cm) &   &  & \quad (in cm)  \\ \midrule
 1.618 & 1.32 & 2.116 & 10.38 \\ \bottomrule
\end{tabular}}
```

### Best and Worst predictions:

In addition to studying the qualitative learning outcomes like average loss, standard deviation of loss, etc., we also visualize images of the best-8 and the worst-8 predictions in Fig. to visually interpret which parameters the model could and couldn't generalize over. From the figure, it is clear that the model is able to generalize over the various foreground and background textures, including bright and dim lighting conditions. However, shadows contributed to some of the higher estimation errors.

## Simulation Results

In this section, we integrate the CNN estimator with a physics-based robot simulator in and the gait-library based controller in to evaluate the closed-loop autonomous operation of the robot. The robot dynamics are simulated in Matlab. At the beginning of each step, robot's current position (specifically the camera position) is supplied to Blender to render a first-person-view synthetic outdoor image using the information from the terrain generator. This image is in turn supplied to the Convolution Neural Network (implemented in Keras and TensorFlow) to predict the step length for the next step. Provided with this predicted step length, the closed-loop dynamics of the robot and controller are simlated for one step to enable the robot to take the step forward. The process is repeated for subsequent steps. A schematic of the numerical simulation pipeline is shown in Fig. .

Numerical simulations are carried out with stones randomly placed with the inter-stone distance within the $[45 : 85]$ cm range. Recall that the camera position and stone location were uniformly sampled from dissimilar ranges in Section IV. The label used for training the CNN is the distance to the camera which is difference of stone location and camera position. Therefore, the distribution of labels used for training is no longer uniform. Accounting for this fact, the step length ranges have been adjusted to only test the CNN in the range where there was enough data to guarantee a good learning outcome.

In this study, we render images with light fixed in an overhead position and focus on carefully examining sensitivity to potential failure modes like camera position or step length going outside the range used for dataset generation during continuous simulation. (We have already noted earlier that shadows are a failure mode.) Our evaluation is based on two metrics, 1) Perception Error and 2) Foot Placement Error. Prediction Error is defined as the difference between Predicted Step Length and True Step Length and it is purely an artifact of the perception module. Similarly, Foot Placement Error is defined as the difference between foot-contact-point and stone-center. This error is the cumulative error of both perception and control.

The robot was simulated for $100$ steps and these two errors are plotted, as shown in Fig. . Note that the prediction error is bounded within a $5$ cm range from the center for the most part and doesn't show significant accumulation of error over time. In Fig. (b), the foot placement error is also plotted for the case where perception was not used (called WOP - With Out Perception) and with perception (using SOD - synthetic outdor dataset) for comparison. In an attempt to minimize effort, the conservative controller always under-steps. In contrast, the prediction model over-estimates more frequently. Therefore, the errors cancel more often and lead to a desirable and mostly-bounded stepping behavior.

# Conclusions and Future Work

In this work, we outlined a systematic way to design a CNN-based predictor that can estimate step lengths for a dynamic bipedal walker operating in discrete terrain. It is shown empirically that a feed-forward gait adjustment based on intermittent visual feedback is sufficient to walk on a discrete terrain where high-speed prediction and accurate foot placement are critical. Several visual factors that impact the predictor's performance are identified for further refinement. This paper integrates gait optimization, nonlinear control, vision, and deep learning.

As part of future work, the direct perception model will be extended to predict other related gait parameters like step width, step height and yaw angle, enabling autonomous rough terrain navigation. Moreover, since multiple steps are visible in an image, accuracy could be improved through Recurrent Neural Networks. Domain adaptation techniques that transfer the predictor from simulation to real world images will enable testing the algorithm on the real robot.

[^1]: This work is supported in part by grants from the Pennsylvania Infrastructure Technology Alliance, the Google Faculty Research Award, and in part by NSF grant IIS-1526515.

[^2]: A. Siravuru, A. Wang, and Q. Nguyen are with the Department of Mechanical Engineering, Carnegie Mellon University, Pittsburgh, PA 15213, USA `{avinashs,allanwan,firstname.lastname@example.com`.

[^3]: K. Sreenath is with the Dept. of Mechanical Engineering, University of California, Berkeley, CA 94720, `firstname.lastname@example.com`.