Connecting Gaze, Scene, and Attention: Generalized Attention Estimation via Joint Modeling of Gaze and Scene Saliency
Eunji Chong Nataniel Ruiz Yongxin Wang Yun Zhang Agata Rozga James Rehg





Connecting Gaze, Scene, and Attention








E. Chong, N. Ruiz, Y. Wang, Y. Zhang, A. Rozga, and J. M. Rehg



School of Interactive Computing, Georgia Institute of Technology, Atlanta, GA, USA

{eunjichong,nataniel.ruiz,ywang751,yzhang467,agata,rehg}@gatech.edu

Connecting Gaze, Scene, and Attention: Generalized Attention Estimation via Joint Modeling of Gaze and Scene Saliency
    Eunji Chong Nataniel Ruiz Yongxin Wang Yun Zhang Agata Rozga James M. Rehg
    October 13, 2018
=====================================================================================================================




This paper addresses the challenging problem of estimating the general visual attention of people in images. Our proposed method is designed to work across multiple naturalistic social scenarios and provides a full picture of the subject's attention and gaze. In contrast, earlier works on gaze and attention estimation have focused on constrained problems in more specific contexts. In particular, our model explicitly represents the gaze direction and handles out-of-frame gaze targets. We leverage three different datasets using a multi-task learning approach. We evaluate our method on widely used benchmarks for single-tasks such as gaze angle estimation and attention-within-an-image, as well as on the new challenging task of generalized visual attention prediction. 
In addition, we have created extended annotations for the MMDB and GazeFollow datasets which are used in our experiments, which we will publicly release.










§ INTRODUCTION

As humans, we are exquisitely sensitive to the gaze of others. We can rapidly infer if another person is making eye contact, follow their gaze to identify their gaze target, categorize quick glances to objects, and even identify when someone is not paying attention <cit.>. Automatically detecting and quantifying these types of visual attention from images and video remains a complex, open challenge. Although gaze estimation has long been an active area of research, most work has focused on relatively constrained versions of the problem in specific predetermined contexts. For example, <cit.> predict the gaze target given that the person is looking at a point on a smartphone screen, <cit.> predicts fixation on an object given that the person is looking at salient object within the frame, <cit.> predict eye contact given that the camera is located near the subject's eyes, and <cit.> predicts the focus of a person's gaze across views in commercial movies which include camera views that follow the actor's attention. It remains a significant challenge to design a system that can model the visual attention of subjects in unconstrained scenarios, without the preconditions utilized by prior works. We call this the problem of generalized visual attention prediction.

The three examples in Figure <ref> illustrate the difficulty of the challenge. In Figure <ref> (a), the subjects are looking at a salient object in the scene, while in (b) the subject is looking somewhere outside of the scene, and (c) they are looking at the camera. The case of Figure <ref> (a) is addressed by the pioneering work of Recasens et al. <cit.>, which tackles this problem by obtaining human annotations of subject gaze targets, leveraging the finding from <cit.> which indicate that annotators very often agree on which object is salient in the scene. 
Their approach, however, was not designed to handle cases (b) and (c) since the dataset annotation process forces human annotators to label a point in the image as the fixation location. In other words the dataset does not distinguish between subjects looking at a point inside of the image or looking somewhere outside of the image. A purely saliency based approach would also fail: notice that there are salient objects in (b), an American flag, and (c), a mug, which can confound such an approach.



Figure <ref> (c) corresponds to the case of screen-based eye tracking <cit.>, in which subjects look at images on a screen and are captured by a calibrated camera that permits the estimation of the gaze location. 

The scenario in Figure <ref> (a) corresponds to gaze following and has been addressed in <cit.>. 
Figure <ref> (b) represents a challenging case, which has not been addressed by prior work, in which the gaze target lies outside the frame and therefore cannot be identified explicitly without additional information.


It is challenging to design an attention model which can deal with these three different scenarios reliably. We tackle this problem by developing a novel generalized visual attention estimation method which jointly learns a subject-dependent saliency map and a 3D gaze vector represented by yaw and pitch. This allows us to estimate the final fixation likelihood map.

Our method is designed so that the fixation likelihood map becomes close to zero when the subject is looking outside the frame, as in cases (b) and (c). When the subject is looking at a target which is visible in the image, as in case (a), then the fixation likelihood map predicts where the subject is likely to be attending. The model simultaneously estimates the 3D gaze angle to provide a complete picture of the subject's attention and gaze. As a result, our approach produces interpretable results spanning all three of the cases in Figure <ref>. 
 
Our Contribution. The major contribution of this work is our method for generalized visual attention prediction, which works across most natural scenarios. To effectively train our model, we exploit three public datasets that were originally collected for different tasks. Specifically, we use the EYEDIAP dataset <cit.> to learn a precise gaze angle representation, a modified version of the GazeFollow dataset <cit.> to learn a gaze-relevant scene saliency representation, and the SynHead dataset <cit.> to complement the first two datasets as it includes large face pose variations and subject attention outside of the image frame.

As a result of our multi-task learning approach, our model achieves state-of-the-art results on the GazeFollow <cit.> task, which consists in identifying the location of the scene the subject is looking at. Our model also competes with state-of-the-art models on the 3D gaze estimation task from the EYEDIAP dataset <cit.>. Most importantly, we evaluate our full model on a new challenging task that automatically quantifies dense visual attention in naturalistic social interactions. We report our results on the Multimodal Dyadic Behavior (MMDB) dataset <cit.>, a dataset of video recordings of the social and communicative behavior of toddlers. This dataset has frame-level annotations of subject's visual targets among many other nonverbal behaviors. We are the first to report attention estimation results on this dataset. We compare our results to several baselines, demonstrating the superior performance of our method.




§ RELATED WORK

Gaze Estimation: Gaze estimation aims to predict the gaze of a human subject. Our work is related to third person gaze estimation and tracking methods which seek to estimate either the three dimensional direction of the gaze or the fixation point of the gaze on a screen. Krafka et al. <cit.> predict the coordinates of the gaze on a smartphone screen and present a dataset which addresses this problem. Mora et al. <cit.> present EYEDIAP, a dataset designed for the evaluation of gaze estimation which was collected in a controlled lab environment. They devise an RGB-D method which predicts the 3D vector of the gaze of the subject. There exist datasets which address similar tasks, such as MPIIGaze <cit.>, as well as synthetic datasets of eye images for gaze estimation <cit.>.
In addition to predicting the 3D gaze vector, our work predicts a fixation likelihood map of the scene as well as whether the person is looking at a location inside or outside of the image.

Visual Saliency: The objective of visual saliency prediction is to estimate locations in an image which attract the attention of humans looking at the image. Since the seminal work of Itti et al. <cit.> visual saliency prediction has been extensively studied. Recently deep learning methods have shown superior performance on this task due to their ability to learn features and to incorporate both local and global context into the prediction <cit.>. Our work in generalized visual attention prediction is influenced by the task of visual saliency since people tend to look at salient objects inside a scene, yet it is distinct because we consider cases where the subject is not looking at any object in the scene. A method driven primarily by saliency detection would not succeed in the latter case. Furthermore, real-world scenes are more likely to generate a wide range of 3D gaze directions in comparison to a screen-based eye-tracking scenario.


Gaze Following: The paper by Recasens et al. <cit.> presents a new computer vision problem which inspired this work. The problem can be described as follows: given a single image containing one or more people, predict the location that each person in the scene is looking at. They presented a novel dataset which contains manual annotations of where the subjects are looking in each image. Our work differs in that we consider cases where the subjects are looking outside of the frame, and we predict the gaze direction in these cases even though the gaze target is not visible. In addition to predicting a fixation likelihood map for the image, we predict the 3D gaze vector of each subject. Gorji and Clark <cit.> study a problem at the intersection of visual saliency and gaze following, which consists of incorporating signals from regions of an image which guide attention to a certain part of the image. For example, when subjects in an image look at an object, this amplifies the apparent saliency of the object. Again, our problem differs in that we do not predict visual saliency but we predict the subject's gaze fixation and gaze direction.

Attention Modeling: Prior works have presented different methods for measuring third-person visual attention using an environment-mounted camera. By assuming body or head orientation is a good proxy for visual orientation, <cit.> projects attention on the street by tracking pedestrians in 3D, <cit.> model the focus of attention in crowded social scene, and <cit.> predicts the object in the scene that a person is interacting with which is usually indicated by hand manipulation or pointing. Our work is certainly related, although it differs because we explicitly consider the gaze of the subject.





§ METHOD



Figure <ref> is an overview of our deep neural network model and its input and output. The model takes three inputs: the whole image, a crop of the subject's face, and the location of her face. Given the input, the model estimates 1. the subject's gaze angle in terms of yaw and pitch degrees (“where” component of visual attention), 2. the subject-dependent saliency in terms of a heatmap (“what” component of visual attention), and 3. how likely the subject is fixating at the estimated gaze target in the scene (overall “strength” of visual attention). 

The model has two fully-convolutional pathways, one connected to the whole image (Figure <ref>-a) and the other connected to the face image (Figure <ref>-b). The reasoning behind having two separate pathways is inspired by the way humans infer another person's visual attention, as first exploited by <cit.>. For example, when we interpret a person's attention from an image, we infer their gaze direction and consider whether there are any salient objects in the image along the estimated direction. Based on this hypothesis, <cit.> connects two independent conv pathways together to learn the heatmap (Figure <ref>-c). We take this approach further and extend their model by explicitly training for the gaze angle (Figure <ref>-d) with a convolutional pathway that is connected to the face image, using a multi-task learning framework. Adding the gaze angle output as an auxiliary task has several advantages, including the additional supervisory signal that we can devise based on the relationship between gaze heatmap and angle, which pushes performance in heatmap estimation even further.

Lastly, we define the likelihood of fixation: a single-valued measure of how likely it is that the subject is looking at the estimated target region inside the frame. It is modeled by a fully connected layer (Figure <ref>-e). Using this last output, the model can produce a much more complete estimation of a person's visual attention. Think of the case of Figure <ref>-b or c, where the person is looking outside the image frame. In such cases, we want the heatmap to be as close to zero as possible since the person is not attending to any point inside of the image. By training this last layer to produce higher value for when it is more certain that the heatmap region is attended to and lower value otherwise, the value can be applied to the heatmap with an operator ⊙ which can be a weighting operator or a gating operator depending on application.

Since there exists no single dataset that covers all of the various gaze and scene combinations that we address in this paper (e.g., looking outside the frame, looking at the camera, fixation on an in-frame object, etc), we adopt a cross domain learning approach where the model learns partial information relevant to each task from different datasets. Depending on what supervisory signal is available in a given batch of training data, the model selectively updates its corresponding branches.

We describe the model architecture in more detail in subsection <ref>. We elaborate on the loss function in <ref> and talk about the datasets and training procedures in <ref>.



 §.§ Model

The inputs given to the model are the entire image, the subject's cropped face and the location of the face of the subject whose attention we want to estimate. The two images are resized to 227×227 so that the face can be observed in higher resolution by the network. Face position is available in terms of the (x,y) full image coordinates. These coordinates are quantized into a 13×13 grid and then flattened to a 169 dimensional 1-hot vector.

The model consists of two convolutional (conv) pathways: a face pathway (Figure <ref>-d) and a scene pathway (Figure <ref>-c). 
ResNet 50 <cit.> is used as a backbone network for the conv pathways (Figure <ref>-a and b). Specifically we use all conv layers of ResNet50 for each of our conv pathways. After each ResNet50 block we add three conv layers (1x1, followed by 3x3, followed by 1x1) with ReLu and batch norm - with stride 1 and no padding. The blue conv layers represented in (c) have filter depth of 512, 128 and 1 respectively. The purple and red conv layers after the face pathway (represented in (c) and (d)) have filter depth of 512, 128 and 16. These conv layers serve to reduce the dimensionality of the features extracted by the ResNet50 backbone networks.

In the face pathway, the feature vector computed with the face input image goes through a fully connected layer to predict the gaze angle represented using yaw and pitch intrinsic Euler angles. In the scene pathway, the feature vectors extracted from the whole image as well as from the face image are concatenated with the face position input vector to learn the person-centric heatmap. Similarly to face position, the ground truth used for learning the heatmap is available as a gaze target position in (x,y) coordinates which is quantized into 10 grids in each dimension.

Lastly, the input vectors to the last layer of each pathway are concatenated and go into the final fully connected layer to estimate the “strength” of the fixation ie. how likely it is that the person is actually fixating at a gaze target within the observable scene. The training label for this value is equal to 1 for a fixation inside of the image and 0 when the subject is looking outside of the scene. We also explore alternative model architectures and restrict our training to a subset of the three datasets. Experiments are reported in Section <ref>.



 §.§ Loss

As our model predicts gaze angle, saliency map and the fixation likelihood, we need to apply appropriate loss functions for each task. For the angle regression task we use an L1 loss, and for the other two tasks we use a cross entropy loss.
Moreover, we recognize that the gaze angle and fixation target predictions are closely related. Based on their relationship additional constraints can be imposed to augment the training loss signal. Namely, when the subject is looking at a target, the actual gaze is a ray from the subject's head to the gaze target. This ray can be projected onto the image. It becomes a 2D vector coming from the subject's head to the target exemplified by the blue vector in Figure <ref>. If the estimated angle is close to the actual one, the projected gaze angle on the image (orange vector in Figure <ref>) should also be close to the blue vector. The proximity of the two vectors is measured using the cosine distance. We call this the project and compare loss.





 §.§ Cross-Domain Datasets and Training Procedure



The largest challenge in training our model is the lack of availability of training examples. Although there are a couple of existing datasets that are suitable for training certain parts of our network, no single dataset contains all of the information that we need to train the full model. Therefore, we leverage three different datasets, namely, GazeFollow <cit.>, EYEDIAP <cit.>, and SynHead <cit.>. We selectively train different sub-parts of our network at a time depending on the available supervisory information within a training batch. See Figure <ref> to see sample images from each dataset.

GazeFollow <cit.> is a real-world image dataset with manual annotations of the locations where people are looking. The images are taken from other major datasets such as MS COCO <cit.> and PASCAL <cit.>. As a result, the images cover a wide range of scenes, people, and gaze directions. However, the actual 3D gaze angles are not available. Furthermore, images where subjects are looking outside of the image frame are not distinguished and all images have a fixation annotation inside of the frame. Although it is mentioned in <cit.> that if the annotators indicated that the person was looking outside the image, the image would be discarded, we notice that there are a considerable number of images in which persons appear to be looking outside of the frame. Therefore, we added additional annotations to this dataset in the form of a binary indicator label for “looking inside” or “looking outside” for every image. In total, we identified 14,564 images correspond to the “looking outside” case which is approximately 11.6% of the total training samples. We have publicly released our additional annotations along with this paper. 

EYEDIAP <cit.> dataset is designed for the evaluation of the gaze estimation task. It has videos of 16 different subjects with full face and background visible in a laboratory environment. Each subject was asked to look at a specific target point on a monitor screen and the 3D gaze angle was annotated by leveraging camera calibration and face depth measurement from depth camera. This dataset contains precise 3D gaze angles for frames where the person is fixating the target point. The dataset also contains video of the subjects looking at a 3D ball target instead of 2D screen target point, but we exclude these ball sessions from our experiments in order to conduct a fairer comparison with prior work. We randomly hold out four subjects for test and use the rest of the sessions for training. Since subjects were looking at a screen, all of the frames can be considered as looking outside the image. However, since the dataset has been collected in a controlled setting the backgrounds are primarily white and there is not a lot of variety in lighting or pose. Also, measured gaze angles range between -40 to 40 which is rather limited.

NVIDIA SynHead <cit.> is a synthetic dataset created for the head pose estimation task. The dataset contains 510,960 frames of 70 head motion tracks rendered using 10 individual head models. The gaze of the head is fixed and aligned with the head pose, thus we use the labeled 3D head pose as the gaze angle ground truth. One of the advantages of a synthetic dataset is the ability to insert different images in the background. We randomly generated 15% from the total frames augmented with provided natural scene backgrounds and regard all as “looking outside” examples. The main reason we include SynHead in training is because it complements the EYEDIAP dataset, as the angle ranges are larger, between -90 and 90, and it can include more diverse backgrounds. Since head pose estimation is not a focus of this paper we do not set aside a test set and use SynHead entirely for training. Dataset details are also summarized in Table <ref>.

Training Procedure. Since each dataset is relevant only to certain subtasks, we only update the relevant parts of the network based on which dataset the training sample is from, while freezing other irrelevant layers during back-propagation. Specifically, when learning gaze angle estimation, we only update the angle pathway (b) and (d) in Figure <ref>, when learning saliency we update the scene pathway (a), (b) and (c) while freezing all other layers. Similarly, when training fixation likelihood we only update the layer (e) in Figure <ref>. We found that this selective back-propagation scheme is critical in achieving good performance. 

In every batch, we draw random samples from all of the datasets shuffled together and perform three separate back-propagation for the three outputs as just described. In the beginning, both convolutional pathways were initialized using a ResNet50 model pre-trained on the ImageNet classification task <cit.>. We use the Adam optimization algorithm with a learning rate of 2.5e-4 and a batch size of 36. Training usually converges within 12 epochs. All of our implementation and experiments are done in PyTorch <cit.>.








§ EVALUATION

In this section we evaluate our model by comparing each output with a number of existing methods and baselines. We first evaluate the person-dependent saliency map in <ref>, gaze angle estimation in <ref> and general attention estimation in <ref>. Lastly, we evaluate our method by changing model architectures and training dataset in <ref>.



 §.§ Person-Dependent Saliency Prediction

We evaluate the performance of saliency map estimation using the suggested test split of the GazeFollow dataset. The test split contains all “looking inside” cases and each test image has multiple gaze target annotations. Following the same evaluation method by <cit.>, we compute the Area Under Curve (AUC) score of the Receiver Operating Characteristic (ROC) curve in which the ground truth target positions are the true labels and heatmap value on corresponding positions are prediction confidence score. Our method achieves a score of 0.896 achieving state-of-the-art performance. Along with AUC we also report results in L2, min distance and angle metric. Please refer to  <cit.> for details about the metric. The numbers are summarized in Table <ref> along with a number of baselines reported in <cit.>. Qualitative results are presented in Figure <ref>. 









 §.§ Gaze Angle Prediction

We report the 3D gaze estimation accuracy based on the yaw and pitch output of our model on the chosen EYEDIAP test split. Table <ref> shows the angular errors in which we achieve less than 0.5 degrees of difference to the state-of-the-art appearance-based gaze estimation method. It is worth noticing that the middle two values come from <cit.> which are computed by five-fold cross validation with the entire EYEDIAP dataset whereas our method is evaluated on a single train/test split. Although we did not choose to perform full cross validation, we conclude that it reaches reasonable accuracy on the benchmark. Note also that our method is trained on multiple tasks whereas all other methods are trained solely on the gaze angle prediction task.







 §.§ Generalized Attention Prediction During Naturalistic Social Interactions

The primary inspiration for our work stems from the need for the ability to quantify various types of visual attention behavior, which is one of the most important nonverbal social cues used in our daily life. Moreover, this is of particular interest among researchers who study child development since gaze behavior of young children is closely related to their social development and developmental disorders such as Autism <cit.>. 
The MMDB dataset is one of the largest datasets that contains children's social and communicative behaviors, collected in order to facilitate data-driven analysis of child behavior based on video. The dataset contains a wide range of nonverbal behavior such as hand gestures, smile, and gaze. It has frame-level human annotations of each behavior. As for gaze, each frame is annotated when the child is looking at a ball, book or the examiner. This is done by human annotation based on multiple views, therefore the child's gaze target can be visible or not depending on the viewpoint. Since the annotation does not indicate in which view the gaze target is visible, we added additional annotation ourselves and identified if the target is visible in a child-facing camera view to construct labels for the general attention estimation problem. We publicly release this annotation text file along with our paper. 

We evaluate our method on the generalized attention prediction task. We design a gaze target grid classification task, where each test image is divided into N×N grids. If the subject is looking inside of the image then the grid square which contains the gaze target is assigned a label of 1 while others are assigned labels of 0. If the subject is looking somewhere outside of the frame then all grid squares are assigned the 0 label. Using our method's fixation likelihood map we predict the positive gaze grid square. We test the GazeFollow model <cit.> which is the closest work to our method in terms of having the ability to predict gaze target location. One of its limitations is the inability to correctly predict the “outside” case, where the subject is looking outside of the frame. As a result, our method achieves much higher precision in addition to increased recall as shown in Table <ref>.

Additionally, we constructed various baseline tests consisting of a classifier based on a subset of features constructed for saliency, gaze and head pose. Specifically, we tested with SVM and Random Forest using a subset of {<cit.>, <cit.>, <cit.>} as features. In other words, each classifier has been trained for detection of looking inside with the training set described in Table <ref>, using aforementioned features, and tested on the MMDB images. 
We report the results in Table <ref>. Note that the MMDB dataset was not used for training across all methods including ours. 
Moreover, we evaluate the value of multi-dataset training in solving the general attention problem. As shown in the last three rows of Table <ref>, joint training of gaze and saliency is critical in solving the general attention estimation task because without the gaze angle estimate it is ineffective to determine whether the subject is looking inside or outside the frame.














 §.§ Alternative Model and Diagnostics

Finally, we run additional experiments to study how the performance of our model is affected by different training datasets and architectural choice by evaluating it on the GazeFollow benchmark. As shown in Table <ref>, omitting EYEDIAP or SynHead training dataset did not have much impact on the attention-within-an-image heatmap estimation whereas changing model architecture considerably affected the scores. For example, using a single ResNet50 pathway which pools facial features using ROI-pooling shows significantly degraded performance which supports our decision to use a scene pathway as well as a face pathway. 
Interestingly, the project-and-compare loss was not as helpful as initially expected, and we think that this is because the coverage range of pose in the SynHead and EYEDIAP datasets is limited (within ± 90) which is not the case in the GazeFollow dataset.

Qualitatively, we were able to observe that, even though our method is designed to measure fixation outside, it can make mistakes when the target is within the frame but occluded by other object. Also, when the subject is closer to the camera than some salient object in the background, the method sometimes estimates those as fixation candidate due to the lack of scene depth understanding. Examples are illustrated in Figure <ref>.











§ CONCLUSION

In this paper we presented the new challenging problem of generalized visual attention prediction which encapsulates several constrained attention prediction and gaze estimation problems that have been the focus of many previous works. We proposed a multi-task learning approach and neural architecture leveraging three different datasets which tackles this problem and works across multiple naturalistic social scenarios. In order to train our architecture we have supplemented these datasets with new annotations and we release these annotations to the public. Our model achieves state-of-the-art performance on the single-task gaze-saliency prediction and competes with state-of-the-art methods on gaze estimation benchmarks while achieving promising performance on the generalized attention prediction problem on the MMDB dataset. Future work in this area can lead to breakthroughs in attention prediction applications which are valuable in numerous scientific and commercial areas. A suggested first step would be to improve existing datasets with additional annotations or collect datasets tailored for this problem.



§ ACKNOWLEDGEMENT

This study was funded in part by the Simons Foundation under grant 247332.







splncs04



