1

Personalized Saliency and Its Prediction

arXiv:1710.03011v1 [] 9 Oct 2017

Yanyu Xu, Junru Wu, Nianyi Li, Shenghua Gao, and Jingyi Yu
Abstract—Almost all existing visual saliency models focus on predicting a universal saliency map across all observers. Yet psychology
studies suggest that visual attention of different observers can vary a lot under some specific circumstances, especially when they view
scenes with multiple salient objects. However, few work explores this visual attention difference probably because of lacking a proper
dataset, as well as complex correlation between visual attention, personal preferences, and image contents.
In this paper, we set out to study this heterogenous visual attention pattern between different observers and build the first dataset for
personalized saliency detection. Further, we propose to decompose a personalized saliency map (referred to as PSM) into a universal
saliency map (referred to as USM) which can be predicted by any existing saliency detection models and a discrepancy between them.
Then personalized saliency detection is casted as the task of discrepancy estimation between PSM and USM. To tackle this task we
propose two solutions: i) The discrepancy estimation for different observers are casted as different but related tasks. Then we feed the
image and its USM into a multi-task convolutional neural network framework to estimate the discrepancy between PSM and USM for
each observer; ii) As the discrepancy is related to both image contents and the observers’ person-specific information, we feed the
image and its associated USM into a convolutional neural network with person-specific information encoded filters to estimate the
discrepancy. Extensive experimental results demonstrate the effectiveness of our models for PSM prediction as well their
generalization capability for unseen observers.
Index Terms—Universal Saliency, Personalized Saliency, Multi-task Learning, Convolutional Neural Networks.

F

1

I NTRODUCTION

ALIENCY refers to a component (object, pixel, person) in a
scene that stands out relative to its neighbors and has been
considered key to human perception and cognition. Traditional
saliency detection techniques attempt to extract the most pertinent
subset of the captured sensory data (RGB images or light fields)
for predicting human visual attention. Applications are numerous,
ranging from compression [1] to image re-targeting [2], and most
recently to virtual reality and augmented reality [3].
By far, almost all previous approaches focus on exploring a
universal saliency model, i.e. , to predict potential salient regions
common to observers while ignoring their differences in gender,
age, personal preferences, etc. Such universal solutions are beneficial in the sense they are able to capture all “potential” saliency
regions. Yet they are insufficient in recognizing heterogeneity
across individuals. Examples in Fig. 1 illustrate that while multiple
objects are deemed highly salient within the same image (e.g. ,
human face (first row), text (last tow rows) and object of (high
color contrast), different observers have very different fixation
preferences when viewing the image. For the rest of the paper, we
use term universal saliency to describe salient regions that incur
high fixations across all observers and term personalized saliency
to describe the heterogeneous ones.
In fact, heterogeneity in saliency preference has been widely
recognized in psychology: “Interestingness is highly subjective
and there are individuals who did not consider any image interesting in some sequence” [4]. Additionally, such inconsistency in
saliency preference has also been well recognized in psychology
and related areas [34] [29] [30]. Therefore, once we know an
observer’s personalized interestingness over each image (personalized saliency), we shall design tailored algorithms to cater to
him/her needs. Personalized saliency detection can contribute to

S

•

Yanyu Xu, Junru Wu, Shenghua Gao and Jingyi Yu are with ShanghaiTech
University, China. Nianyi Li is with University of Delaware, United States.

Fig. 1: An illustration of PSM dataset. Our dataset provides both
eye fixations of different subjects and semantic labels. Due to
the large amount of objects in our dataset, for each image, we
didn’t fully segment it and only labelled objects that cover at
least three gaze points from each individual. A notable difference
between PSM and its predecessors is that each subjects looks 4
times on PSM data to derive solid fixation ground truth maps.
Both commonality and distinctiveness exist for PSMs viewed by
different participant. This motivates us to model PSM based on
USM.

various applications in future computer vision tasks. For example,
in the application of image retargeting, the texts on the table in
the fourth row in Fig. 1 should be preserved for observer B and C
when resizing the image whereas such texts are less important for
observer A. In VR/AR applications, how to effectively compress
the data stream is a crucial problem. Once the personal gaze
pattern is available, we can realize user-correlative compression
algorithm, which enables to keep the scene reality at minimal
size for different users. Another straightforward application is

2

in advertisement deployment. The location of the advertising
window could be adjusted according to the his/her personalized
gaze pattern in devices such as VR helmet.
Nevertheless, the difficulty in gathering sufficient gaze information and lacking of industry support to build a large enough
personalized saliency detection dataset stops researchers from
exploring the personalized gaze pattern problem. The recent
emerging of VR/AR techniques push the needs of gathering userrelated data on board. For this reason, we construct the first
personalized saliency detection dataset, which consists of 1600
images viewed by 30 observers. To improve reliability, each
image is viewed by each observer 4 times in different period
of time within a week. We use an ‘Eyegaze Edge’ eye tracker
to record individuals’ gaze patterns and produce a set of 48,000
(1, 600 × 30) eye fixation maps. The total time for data collection
procedure is about 255 hours. To correlate the acquired PSMs
and the image contents, we manually segment each image into
a collection of objects and semantically label them. Examples in
Fig. 1 illustrate how fixations vary across three human subjects.
Our annotated dataset provides fine-grained semantic analysis for
studying saliency variations across individuals. For example, we
observed that certain types of objects such as watches, belts would
introduce more incongruity (possibly due to gender differences)
whereas other types such as faces would lead to more coherent
fixation maps, as shown in Tab. 2.
In this paper, we present a computational model towards this
personalized saliency detection problem. As shown in Fig. 1,
saliency maps from different observers still share certain commonality, which is encoded in USM. Hence, we propose to model the
PSM as a combination of USM and a discrepancy map which is
related to the observer’s personal preferences and the image contents. To model this discrepancy map, we propose two solutions:
i) We adopt a multi-task convolutional neural network (CNN) to
identify the discrepancy between PSM and USM for each person,
as shown in Fig. 4; ii) motivated by the findings in psychology
[36] [34] [29] [30] that such discrepancy is closely related to the
observer’s personal information, including, gender, age, personal
preferences, etc., we propose to encode the personal information
into the CNN filters for discrepancy prediction, and term such
solution as CNN with person-specific information encoded filters
(CNN-PIEF), as shown in Fig. 5.
To sum up, the contributions of this paper are summarized as
follows:
1) To the best of our knowledge, it is the first work that specifically tackles the personalized saliency and we build the first
dataset for personalized saliency detection. It is also worth
noting that both the personal information of each observer as
well as the object segmentation annotations are also provided
in our dataset;
2) We propose to model PSM as the combination of USM and
the discrepancy between PSM and USM. Experimental results
demonstrate the effectiveness of our framework;
3) We propose to use multi-task CNN and CNN-PIEF to predict
the discrepancy between PSM and USM. Compared with
multi-task CNN, CNN-PIEF can be easily generalized to those
observers who are not in the training set, thus it is more
scalable in open-set setting.
A preliminary version of this work was presented at [51]. We
have extended our work in the following aspects: i), we enlarge the
Personalized Saliency Dataset to 30 observers, and the personal
information of each observer, including gender, the major of the

observer, personal preferences to sports, colors, etc. have been
included in our dataset; ii), we propose to leverage a CNN with
person-specific information encoded filters (CNN-PIEF) to predict
the discrepancy map between PSM and USM, which is more
scalable for open-set setting; iii) more measurements and more
experiments are conducted to analyze our dataset and evaluate our
proposed method for PSM prediction.
The rest of this paper is organized as follows: In Section 2,
we will review related methods and datasets used for universal
saliency detection. In Section 3, we will introduce and analyze
our personalized saliency detection dataset. In section 4, we
will detail our solution for personalized saliency detection. We
will experimentally evaluate our proposed personalized saliency
detection method in Section 5 and conclude our work in Section
6.

2

R ELATED W ORK

Tremendous efforts on saliency detection have been focused on
predicting universal saliency. In this section, we will briefly discuss the most relevant ones, including both hand-crafted features
based and deep learning based saliency detection methods. We
refer the readers to [5] for a comprehensive study on existing
universal saliency detection schemes.

2.1

Universal Saliency Detection Benchmarks.

There are a few widely used saliency object detection and fixation
prediction datasets, in which each image is generally associated
with a single ground truth saliency map, averaged across the
fixation maps across the participates. To select images suitable
for personalized saliency, we explore several popular eye fixation
datasets. The MIT dataset [6] contains 1,003 images viewed by
15 subjects. In addition, the PASCAL-S [7] dataset provides the
ground truth for both eye fixation and object detection and consist
of 850 images viewed by 8 subjects. The iSUN dataset [52],
a large scale dataset used for eye fixation prediction, contains
20,608 images from the SUN database. The images are completely
annotated and are viewed by users. Finally, the SALICON dataset
[9] consists of 10,000 images with rich contextual information.

2.2
2.2.1

Universal Saliency Detection
Hand-crafted Features Based Saliency Detection.

The work of [46], one of the classic saliency models, used
the predefined color subspaces such as intensity, red-green, and
blue-yellow color opponencies and four orientations to represent
image and simulate the receptive fields of various neurons through
local center-surround differences to predict saliency maps. [47]
employed a intra and inter channel fusion strategy for color and
orientation feature channels with multi-scale rarities to predict
saliency map. [48] simulated the saliency through calculating the
distance between low level features including color, orientation,
and spatial features extracted at a local image patch. [50] used
high-level features extracted from person and face detectors to
address this saliency task. [49] investigated the effect of highlevel features including the presence of text and cars in estimating
saliency maps.

3

2.2.2

CNN Based Saliency Detection.

Our personalized saliency detection exploits convolutional neural
network (CNN) in light of its great success in multiple computer
vision tasks, e.g. , image classification [40], semantic segmentation [41], as well as saliency detection [42] [43] [12] [13].
An early approach of Ensembles of Deep Networks (eDN) [42]
was proposed by Vig et al. , where feature maps from different
layers in a 3-layers ConvNet are fed into a simple linear classifier
for the salient or non-salient classification. Later DeepVisual
attention proposed by Kummerer et al. [43] leverages AlexNet
[40] trained for image classification to extract features for eye
fixation regression. In a similar vein, Huang et al. [9] proposed to
fine-tune CNNs pre-trained for object recognition (AlexNet [40],
VGG-16 [44] and GoogLeNet [45]) via a new objective function
based on the saliency evaluation metrics e.g. Normalized Scanpath
Saliency (NSS), Similarity, and KL-Divergence. Pan et al. [10]
proposed to use a shallow CNN trained from scratch and another
deep CNN where the weights of its first 3 layers was adapted
from VGG CNN M trained for image classification. Liu et al.
[27] proposed a multi-resolution CNNs where three final fully
connected layers are combined to form the final saliency map.
Srinivas et al. presented a DeepFix [12] network by using Location
Biased Convolution filters to allow the network to exploit location
dependent patterns. Kruthiventi et al. [13] proposed a unified
framework to predict eye fixation and segment salient objects. All
these approaches have focused on the universal saliency model
and we show many merits of these techniques can be used for
personalized saliency.

3

PSM DATASET

We start with constructing a dataset suitable for personalized
saliency analysis, which is released to the computer vision community for research propose1 .
3.1

Data Collection

Clearly, the rule of thumb for preparing such a dataset is to choose
images that yield distinctive fixation map among different persons.
To do so, we first analyze existing datasets. A majority of existing
eye fixation datasets provide the one-time visual attention tracking
results of each individual human observer. Specifically, we can
correlate the level of agreement across different observers with
respect to the number of object categories in the image. When
an image contains few objects, we observe that an observer tends
to fix his/her visual attention at locations where objects that have
specific semantic meanings, e.g., faces, text, signs [6], [14]. These
objects indeed attract more attention and hence are deemed more
salient. However, when an image consists of multiple objects all
with strong saliency as shown in Fig. 1, we observe an observer
tends to diverge his/her attention. In fact, the observer focuses
attention on objects that attract his/her most personally. We
therefore deliberately choose 1,600 images with multiple semantic
annotations to construct our dataset for PSM purpose. Among
them, 1,100 images are chosen from existing saliency detection
datasets including SALICON [15], ImageNet [16], iSUN [52],
OSIE [14], PASCAL-S [7], 125 images are captured by ourselves,
and 375 images are gathered from the Internet.
1. GitHub: https://github.com/xuyanyu-shh/Personalized-Saliency

PHQERZWLH
ZRPHQERZWLH
PHQKDQGZDWFK
ZRPHQKDQGZDWFK
YLROLQ
NH\ERDUG
RYHQ
ILVKWDQN
FDQ
PHQIDFH
ZRPHQIDFH

3HUVRQ












3HUVRQ












3HUVRQ












3HUVRQ












3HUVRQ












3HUVRQ












3HUVRQ












Fig. 2: The distribution of the interestingness of various objects
for a same participant. The value is calculated as follows: we sum
values of the fixation map intersecting with the mask of a specific
object, and divide it with the total of fixation maps over the whole
image. Thus higher value indicates that the participant puts more
attention on the object.

3.2

Ground Truth Annotation

To gather the ground truth, we have recruited 30 student participants (14 males, 16 females, aged between 20 and 25). All
participants have normal or corrected-to-normal vision. In our
setup, each observer sits about 40 inches in front of a 24inches LCD monitor of a 1920 × 1080 resolution. All images
are resized to the same resolution. We conduct all experiments in
an empty and semi-dark room, with only one standby assistant.
An eye tracker (‘Eyevisual attention Edge’ eye tracker) records
their visual attentions as they view each image for 3 seconds. We
partition 1,600 images into 34 sessions each containing 40 to 55
images. Each session lasts about 3 minutes followed by a half
minute break. The eye tracker is re-calibrated at the beginning of
each session. To ensure the veracity of the fixation map of each
individual as well as to remove outliers, we have each image be
viewed by each observer 4 times. We then combine the 4 saliency
maps of the same image viewed by the same person, and use
the result as the ground truth PSM of the observer. To obtain a
continuous saliency map of an image from the raw data of eye
tracker, we follow [6] by smoothing the fixation locations via
Gaussian blurs.
To further analyze the causes of saliency heterogeneity, we
conduct the semantic segmentation for all 1,600 images via the
open annotation tool LabelMe [17]. Specifically, we annotate
26,100 objects of 242 classes in total and identify objects that
attract more attention for each individual participant. To achieve
this, we compare the fixation map with the mask of a specific
object and use the result as the attention value of the corresponding
object. We then average the result over all images that containing
the same object, and use it to measure the interestingness of
the object to a specific participant. In Fig. 2, we illustrate some
representative objects and persons and show the distribution of
the interestingness of various objects for a same participant. We
observe that all participants exhibit a similar level of interestingness measure on faces where they exhibit different interestingness
measures on various objects such as watch, bow tie, et al. This
validates that it is necessary to choose images with multiple
objects to build our PSM data.
3.3

Dataset Analysis

We will show that our dataset is more suitable for personalized
saliency than the existing ones for several reasons. First, we show
that it is necessary for a participant to view the image multiple

4

Datasets
MIT
OSIE
Ours

Metric value

100
95
90
85

CC
Similarity
AUC−Judd

80
75
1

2

3

4

5

6

7

8

9

Fig. 3: The point with x = n measures the differences between
ground truth saliency maps generated by viewing the same image
n times and n+1 times. This figure shows that when n ≥4, the
ground truth saliency map generated by viewing the image n times
has little difference with that generated by observing the image
n + 1 times. Thus viewing each image 4 times is enough to get a
robust estimation of the PSM ground truth.

times. Then, heterogeneity in saliency maps can be severe. We
explain why not using other existing datasets in the end.
3.3.1 Why Is Each Image Viewed Multiple Times for
Ground-truth Annotation?
To validate whether it is necessity for an observer to view each
image multiple times, we randomly sample 220 images, and each
image is viewed by the same participant 10 times. The time
interval for the same person to view the same image ranges from
one day to one week because we want to get the short term
memory of the person for the given image.
We then calculate the differences of these saliency maps in
terms of the commonly used metrics for saliency detection [18]:
CC, Similarity, and AUC-Judd. We average these criteria for
all persons and all images, and we show the results in Fig. 3.
We observe that the saliency map obtained by viewing each
image only once vs. multiple times exhibit significant differences.
Further, the saliency map averaged over 4 or more times is closer
to the long term result.
3.3.2 Heterogeneity Among Different Datasets.
To further illustrate that our proposed dataset is appropriate for
personalized saliency detection task, we compare the inter-subject
consistency, i.e. , the agreement among different viewers, in our
PSM dataset and other related datasets. Specifically, for each
dataset, we first enumerate all possible subject-pairs, i.e. , two
different subjects, and then compute the average CC, Similarity,
AUC-Judd, and NSS scores across all pairs. Recall that our PSM
dataset consists of images from different datasets, e.g. , MIT,
OSIE, ImageNet, PASCAL-S, SALICON, iSUN etc. , and only
MIT, OSIE, PASCAL-S are designed for saliency tasks2 . Hence,
we only compare the consistency scores among ours and above
three datasets, and we show the results in Fig. 1. We observe that
our dataset achieves the lowest inter-subject consistency values
among all relative ones, indicating that the heterogeneity in our
saliency maps are more severe than the others.
2. Even though SALICON and iSUN are also saliency fixation datasets,
the ground truth were annotated based on mouse-tracking and web camera
respectively.

CC
53.08
51.39
44.56

Similarity
45.85
43.22
41.44

AUC-Judd
89.34
88.78
80.98

NSS
2.1442
2.7745
1.7843

TABLE 1: Inter-subject consistency of different datasets. To compute the inter-subject consistency, we compute CC, Similarity,
AUC-Judd, and NSS for pair-wise saliency maps viewed by
different observers for each image, then we average the results
over all images. For fair comparison, the CC, Similarity, AUCJudd, and NSS of our method reported here is based on the
saliency maps viewed by each observer once.

3.3.3

Why Not Using Existing Datasets?

Firstly, our objective is to identify the heterogeneity of fixation
maps across individuals, so it is crucial for us to ensure reliable
ground-truth annotation. Existing datasets such as MIT, OSIE do
provide individual fixation information, however, each image was
viewed once which cannot lead to robust ground-truth. In section
3.3 and Figure 3, we empirically show that viewing images multiple times leads to a more reliable ground truth, thus in our data
set, each image is viewed 4 times with an one-day time interval.
Secondly, as aforementioned, since fine-tuning a separate network
for each participant is not scalable in real applications, we leverage
person-specific information for personalized saliency prediction.
However, such person-specific information is not available in all
existing datasets.

4
4.1

A PPROACH
USM based PSM Formulation

Many existing work [19] [10] has employed CNN in an end-toend strategy to predict saliency map, and achieves state-of-theart performance. Intuitively, we can follow the same strategy for
PSM prediction, i.e. training a separate CNN for each participant
to map the RGB images to PSM. However, such strategy is
neither scalable nor feasible for a number of reasons. Firstly, it
needs a vast amount of training samples to learn a robust CNN
for each participant. This requires subjects to view thousands
of images with high concentration, which is hard and extremely
time consuming. Secondly, training multiple CNNs for different
subjects is computationally expensive and inefficient.
While each participant is unique in terms of their gender, age,
personal preference, etc. , resulting in their incongruity in saliency
preference, different participants still share commonalities in their
observed saliency maps because certain objects, such as faces and
logos, always seem to attract the attention of all participants as
shown in Fig. 1.
For this reason, instead of predicting the PSM directly, we set
out to explore the difference map between USM and PSM. The
discrepancy map ∆(Pn , Ii ) for the given image Ii (i = 1, . . . , K )
of the n-th participant Pn (n = 1, . . . , N ) is of the form:

SP SM (Pn , Ii ) = SU SM (Ii ) + ∆(Pn , Ii )

(1)

where, SP SM (Pn , Ii ) is the desired personalized saliency
map and SU SM (Ii ) is the universal saliency map.
Note that USMs predicted by traditional saliency method entail
the commonality observed by different participants. We convert
the problem of predicting PSMs to estimating the discrepancy

5

Fig. 4: The pipeline of our Multi-task CNN based PSM prediction. The input is the image with its predicted USM. Specifically, we
treat the discrepancy prediction for each person as a separate task. There is n person, then there is n tasks in this framework. We then
sum the predicted discrepancy map with USM and generate the final estimated PSM.

∆(Pn , Ii ) because that universal saliency map SU SM (Ii ) itself already provides a rough estimation of the PSM, and the
discrepancy ∆(Pn , Ii ) works as an error correction function.
Previous work has shown that error correction scheme works well
for CNN based regression task [31] and classification task [40],
which motivates us to model PSM based on USM. We further
experimentally show that such error correction strategy is much
more efficient than directly estimating PSMs from RGB images,
as shown in Tab. 2.
4.2

Discrepancy Prediction

On one hand, such ∆(Pn , Ii ) is closely related to the contents of
input image; On the other hand, ∆(Pn , Ii ) is subject-dependent.
Previous work observes that human attention is related to several
observer-dependent factors. For example, preference [32] [33],
gender [34], age [35] are all closely related to individual’s visual
attention. Specifically, there is a study showing that female observers would pay more attention on clothes and shoes than male
[36]. Therefore, in this paper, we propose two solutions to predict
such ∆(Pn , Ii ): i) in the case that we don’t know the factors or
the observer’s personal information that results in the PSM, we
use a Multi-task CNN to learn a separate model for discrepancy
prediction a separate observer; ii) If we have the observer’s
personal information that related to personalized saliency over
different images, we encode his/her personal information with
CNN filters, and propose a CNN with Person-specific Information
Encoded Filters (CNN-PIEF) for Discrepancy Prediction. Next
we will detail these solutions, and their advantages/disadvantages
separately.
4.2.1 Multi-task CNN based Discrepancy Prediction
Previous approaches [21] [22] have shown that features extracted
by the first several layers can be shared by related tasks. Actually
the discrepancy prediction are distinct but related regression tasks
across different observers, which motivates our multi-task CNN
solution. In our multi-task CNN based discrepancy prediction,
the discrepancy prediction for different observers corresponds to
different tasks. The inputs of multi-task CNN network are images
with their corresponding universal saliency map and our goal is

to estimate the discrepancy maps ∆(Pn , Ii ) for n-th participants
through n-th task. The network architecture of our Multi-task
CNN is illustrated in Fig. 4.
Suppose we have N participants in total. We concatenate a
160 × 120 resolution RGB image with its USM from general
saliency models and generate a 160 × 120 × 4 cube as the input
of the multi-task network. For image Ii , ∆(Pn , Ii ) is the output
of the n-th task corresponding to the discrepancy between PSM
and USM for the n-th person. There are four convolutional layers
shared by all participants after which the network is then split into
N tasks which is exclusive for N participants. Each task has three
convolutional layers followed by an ReLU activation function.
[19] and [20] show that by adding the supervision in the
middle layers, the features learned by CNN will be more discriminative and can boost the performance of an given task.
Consequently, we set an additional Loss Layer on conv5 and conv6
layer of the n-th task to impose the middle layer supervision ,
which can help the prediction of ∆(Pn , Ii ). For the n-th task,
f`n (SU SM (Ii ), Ii ) ∈ Rh` ×w` ×d` (` = 5, 6, 7) is the feature map
after the `-th convolutional layer (the first convolutional layer corresponds to the first exclusive convolutional layer, so ` starts from
5). For each feature map f`n (SU SM (Ii ), Ii ), a 1×1 convolutional
layer was employed to map it to S` (SU SM (Ii ), Ii ) ∈ Rh` ×w` ×1 ,
which is the target discrepancy. To make S` (SU SM (Ii ), Ii ) close
to ∆` (Pn , Ii ), we set the objective function as:

min

7 X
N X
K
X

kSk (SU SM (Ii ), Ii ) − ∆` (Pn , Ii )k2F

(2)

`=5 n=1 i=1

Then we use mini-batch based stochastic gradient descent to
optimize all parameters in our Multi-task CNN.
Remarks: Multi-task CNN based discrepancy prediction
learns separate but related models for different observers, and
different models implicitly encode the personal-information which
is related to PSM prediction. Compared with CNN-PIEF, we don’t
need to exploit which information are related to PSM prediction.
But in the deployment stage, given any unrecorded observer,
he/she has to view lots of images to get the PSMs. Then we use
these PSMs to retrain the network, which is time-consuming.

6

Fig. 5: The pipeline of our PSM prediction model. The input is the image with its predicted USM. Specifically, we embed the personspecific information into CNN by encoding into filters and then convolving with the outputs of conv4 layer. We then sum the predicted
discrepancy map with USM and generate the final estimated PSM.

4.2.2 CNN with Person-specific Information Encoded Filters for Discrepancy Prediction
Since ∆(Pn , Ii ) is observer-dependent and also relying on the
content of the input image, we propose to modify the network
from [31] and use person-specific information to generate filters
to convolve with the input feature maps, and these person-specific
information encoded filters (PIEF) function as “switches” to determine which areas are interested for different observers. The input
of network are images with their corresponding universal saliency
map and the observer-dependent PIEF, the goal is to estimate the
difference/discrepancy maps ∆(Pn , Ii ) for each participants.
For each individual, we conduct a simple survey to acquire
their personal information, and then encode the collected information to feature vectors by one-hot code. The information of each
observer then are coded by vectors with the same length. Inspired
by the recent work of [37] [38] [39], we adopt a similar way
to embed our person-specific information into CNN, i.e. we then
reshape the features in the last layer into a set of convolutional
filters (4D tensor) which is used to convolve with the outputs from
conv4 layer in our CNN.
Suppose that we have N participants in total, for image Ii ,
the output of CNN-PIEF corresponds to the discrepancy between
PSM and USM for the n-th person: ∆(P, Ii ). We concatenate a
160×120 resolution RGB image with its USM predicted by some
existing saliency model, resulting in a 160 × 120 × 4 cube as
the input of CNN. We use f` (SU SM (Ii ), Ii ) ∈ Rh` ×w` ×d` (` =
5, 6, 7) to donate the feature map after the `-th convolutional layer.
Inspired by previous work [19], [20], middle layer supervision is
imposed by additional Loss Layer on conv5 and conv6 layer. For
each feature map f` (SU SM (Ii ), Ii ), we use a 1 × 1 convolutional
layer to map it to a feature map S` (SU SM (Ii ), Ii ) ∈ Rh` ×w` ×1
which corresponds to the predicted discrepancy. It is desirable
that S` (SU SM (Ii ), Ii ) is close to ∆` (P, Ii ) which is obtained by
resizing to ∆` (Pn , Ii ) to the size of h` × w` × 1. Thus we arrive
at the following objective function:

min

7 X
K
X

kSk (SU SM (Ii ), Ii ) − ∆` (P, Ii )k2F

(3)

`=5 i=1

The network architecture of our CNN-PIEF is illustrated in Fig. 5.

Person-specific Information Collection
Recall that PSM prediction task is challenging because that
too many factors contribute to the saliency variations across
individual. And the investigation of key factors leading to the
observer-dependent factors falls into the regime of psychology.
In this paper, we don’t aim to find out all factors that facilitate the
attention discrepancy. Instead, the goal of our work is to show that
by encoding a fraction of personal information of participants into
CNN, we can approximately model the visual attention pattern of
individual (personalized saliency).
The personal information is comprised of the following parts:
Firstly, previous studies have shown that gender [36] causes the
heterogeneity of saliency map across different observers 3 , which
can also be observed in our experiments. We therefore collect the
gender information observers. Secondly, considering that color
is important factor in universal saliency detection, and our conjecture that different preference/sensitive of different observers
to different colors may cause the inconsistence of saliency, we
also include collect the the preference of different observers over
limited numbers of colors. Specifically, we choose three primary
colors (i.e. Red, Green and Blue), additive secondary colors (i.e.
Cyan, Magenta and Yellow), Black and White, and collect the
preference/disgust of all subjects over these colors. Thirdly we
conjecture that the education background and the hobbies of
different observers may also result in their different interestingness
to different objects. Therefore, we rank all the objects by their
observer-dependency attention variance on the training set, and
pick the top ranked ones to generate PIEF.
In order to calculate the distribution of the interestingness of
various objects to different participants, we manually annotate all
the objects (242 classes in total) in our dataset. Mathematically,
for the nth observer Pn (n = 1, . . . , N ), given the image Ii
(i = 1, . . . , K ), we donate the personalized saliency maps (PSM)
of the nth participant on Ii as SP SM (Pn , Ii ), and donate the
binary mask of a given class Cj (j = 1, . . . , 242) in image Ii as
3. Psychology studies [36] also show that age is the factor cause the heterogeneity of saliency map across different observers, as the age of participants
in our study falls in a small range of 20-25 years old, we don’t include it for
PIEF.

7

MCj ,Ii , we use Int(Pn , Ii , Cj ) to represent the interestingness of
class Cj to participant Pn in image Ii , which is calculated as:
Int(Pn , Ii , Cj,Ii ) =

kvec(SP SM (Pn , Ii ) MCj ,Ii )k1
kvec(SP SM (Pn , Ii ))k1

(4)

We then define the interestingness of class Cj to observer Pn
as Int(Pn , Cj ), which can be calculated as follows:
1 X
Int(Pn , Ii , Cj )
(5)
Int(Pn , Cj ) =
Z I
i

where Z is the number of images which contain the annotated
objects in class Cj . Int(Pn , Cj ) measures the interestingness
of the segment from class Cj to the observer Pn . We rank all
the objects by their observer-dependency attention variance on
the training set. The objects with large attention variance can be
roughly categorized as fashions, sports, texts, etc. as shown in
Fig. 2.
Then we a conduct survey to collect each observer’s personal
information. Specifically, we collect observer’s gender information
(1D), the preference to objects fallen into the fashion category
(ring, necklace, bracelet, earring, hairpin, watch, glasses, tie, belt,
kneelet etc. , 11D), the preference/disgust to colors (red, yellow,
green, cyan, blue, purple, white, black, 16D), the preference
to different sports (football, basketball, badminton, tabletennis,
tennis, volleyball, baseball, and billiards etc. , 11D), and the
preference to other objects fallen into other categories (IT, plant,
texts, food, 4D). One-hot encoding strategy is used to encode each
observer’s personal information. The dimensionality of encoded
information vector for each observer is 43D. Then we feed such
personal-information into CNN-PIEF for PSM prediction.
Remarks: On one hand, the performance of CNN-PIEF depends on the choice of person-specific information, while factors
caused personalized saliency are still not clear in psychology,
which restricts the performance of our model. However, our
experiments show that even with a small fraction of personal information for PIEF, our model still achieves comparable performance
compared with multi-task CNN based discrepancy prediction, as
discussed in Section 5.2; On the other hand,in our CNN-PIEF,
all individuals share the same network. Therefore, in deployment
stage, given any unrecorded observer, our model requires only for
his/her person-specific information to output the PSM. The high
scalability property of our network makes it easy to fit our network
in real applications, as discussed in Section 5.3.

5
5.1

middle layer supervision is imposed by adding 1 × 1 conv layer
after conv5 and conv6; iii) a channel corresponding to USM is
added in the input.
In CNN-PIEF, we trained our network with the following
hyper-parameters: mini-batch size (40), learning rate (1e-6),
momentum (0.9), weight decay (0.0005), number of iterations
(100,000). The parameters are initialized with ‘xavier’.
Evaluation Protocols. We evaluate our approach under two
settings: closed-set setting and open-set setting. Specifically, we
randomly choose 20 observers whose PSMs are used for closedset evaluation, while the PSMs corresponding to the remaining 10
observers are used for open-set evaluation. Of all 1600 images, we
randomly choose 1000 images and use them as training images,
and use the remaining 600 images as testing images. The closedset setting is used to valuate how well our model can predict
the visual attention pattern of seen observers. Therefore we use
the 1000 images corresponding to those 20 observers to train a
model, and evaluate the model with the remaining 600 images
associated with these 20 observers. The open-set setting is used
to evaluate whether our model is transferable, i.e. predicting the
fixation maps for an unseen observer. In the open-set setting,
we evaluate the model trained in closed-set setting with the 600
images corresponding to those 10 unseen objects in the open-set.
Measurements. By following [27] [10] [13], we choose CC,
Similarity, AUC-Judd, and NSS [18] to measure the differences
between the predicted PSM and its corresponding ground truth.
Baselines. In our model, all existing fixation prediction methods can be used to generate universal saliency maps, we then
concatenate it with RGB image as the input of our network. Based
on the performance of existing methods on the MIT saliency
benchmark [24] in terms of similarity, we choose LDS [25],
BMS [26], ML-Net [19], and SalNet [10] to predict the universal
saliency maps on our dataset. The first two methods are based
on hand-crafted features, and the latter two are based on deep
learning techniques. We use the codes provided by the authors of
these methods to generate USMs.
To validate the effectiveness of our model, we compare our
scheme with the following baseline algorithms:
•

•

E XPERIMENTS

•

Experimental Setup

Parameters. We implement our solutions with the CAFFE framework [23]. To avoid over-fitting and improve model robustness,
we augment the training data through left-right flip operations.
In multi-task CNN network, we train our network with the
following hyper-parameters setting: mini-batch size (40), learning rate (0.0003), momentum (0.9), weight decay (0.0005), and
number of iterations (40,000). The parameters corresponding to
the universal saliency map channel and 1 × 1 conv layers for
middle layer supervision are initialized with ‘xavier’. Following
the same initialization step in [10] and [13], we use the welltrained DeepNet model to initialize the corresponding parameters
in our network. The network architecture of our Multi-task CNN
is identical to that of DeepNet [10] except that i) the parameters
corresponding to tasks of different observers are different; ii)

RGB based MultiConvNets: Different ConvNets are
trained to predict ∆(Pn , Ii ) for different observer independently, with RGB images as input.
RGB based Multi-task: Multi-task CNN architecture is
trained to predict ∆(Pn , Ii ) for all participants simultaneously, with RGB images as input.
X based MultiConvNets: Different ConvNets are trained
to predict ∆(Pn , Ii ) for different observer independently,
with RGB images and USM predicted by method X as
input, where X donates LDS, BMS, ML-Net, and SalNet
respectively.

It is worth noting that the network architectures of the baselines are similar with ours. The only differences are the number
of input channels and whether the parameters are shared in the
first few layers. For fair comparisons, we have employed the same
strategies on data augmentation, middle layers supervision, and
parameter initializations for all these baselines.
5.2

Performance Evaluation Under Closed-Set Setting

The performance of all methods under closed-set are listed in
Tab. 2. We also show some predicted saliency maps for different participants in Fig. 7. Our solutions always outperform the

8

5

10

15

0.78
0.76
0.74
0.72
0

20

5

CC

10

15

0.9
0.85
0.8
0.75
0

20

2.5

MultiConvNets
Multi−Task CNN
PIEF−CNN

0.95

Metric value

0.6

0.8

Metric value

Metric value

Metric value

0.7

0.5
0

MultiConvNets
Multi−Task CNN
PIEF−CNN

0.82

MultiConvNets
Multi−Task CNN
PIEF−CNN

0.8

5

Similarity

10

15

2
1.5
1
0

20

MultiConvNets
Multi−Task CNN
PIEF−CNN

5

AUC−Judd

10

15

20

NSS

0.4
5

10

15

0.8

0.7
0

20

5

CC

10

15

0.9

5

Similarity

10

15

MultiConvNets
Multi−Task CNN
PIEF−CNN

2.2

0.8

0.7
0

20

MultiConvNets
Multi−Task CNN
PIEF−CNN

Metric value

0.6

0

MultiConvNets
Multi−Task CNN
PIEF−CNN

0.9

Metric value

0.8

Metric value

Metric value

(a) The improvement of our LDS based Multi-task CNN and LDS based CNN-PIEF
MultiConvNets
Multi−Task CNN
PIEF−CNN

1.8
1.4
1
0.6
0

20

5

AUC−Judd

10

15

20

NSS

0.5
0

5

10

15

0.78

0.74
0

20

5

CC

10

15

MultiConvNets
Multi−Task CNN
PIEF−CNN

0.95
0.9
0.85
0.8
0.75
0

20

5

Similarity

10

15

MultiConvNets
Multi−Task CNN
PIEF−CNN

2.4

Metric value

0.6

MultiConvNets
Multi−Task CNN
PIEF−CNN

0.82

Metric value

0.7

Metric value

Metric value

(b) The improvement of our ML-Net based Multi-task CNN and ML-Net based CNN-PIEF
MultiConvNets
Multi−Task CNN
PIEF−CNN

2
1.6
1.2
0.8
0

20

5

AUC−Judd

10

15

20

NSS

(c) The improvement of our BMS based Multi-task CNN and BMS based CNN-PIEF

0.7

0.6
0

5

10

CC

15

20

MultiConvNets
Multi−Task CNN
PIEF−CNN

0.76
0.74
0

5

10

15

20

Similarity

2.5

MultiConvNets
Multi−Task CNN
PIEF−CNN

0.95
0.9

Metric value

Metric value

Metric value

0.8
0.78

Metric value

MultiConvNets
Multi−Task CNN
PIEF−CNN

0.8

0.85
0.8
0.75
0

5

10

15

20

MultiConvNets
Multi−Task CNN
PIEF−CNN

2

1.5
0

AUC−Judd

5

10

15

20

NSS

(d) The improvement of our SalNet based Multi-task CNN and SalNet based CNN-PIEF
Fig. 6: MultiConvNets vs. our methods for PSM predictions for each observer. In each graph, x axis represents the ID of each
observer and y axis represents the metric value of CC, Similarity, AUC-Judd or NSS. We can see that our methods always outperform
MultiConvNets under closed-set setting.

baseline methods in terms of all metrics, which demonstrates the
effectiveness of our method. Furthermore, the discrepancy based
personalized saliency detection methods consistently outperform
directly predicting PSM from RGB images. This validates the
effectiveness of our “error correction” strategy for personalized
saliency detection.
We further show the performance of different PSM prediction
methods for each observer on the testing set in Fig. 6. Both Tab. 2
and Fig. 6 show that our multi-task CNN scheme and CNN-PIEF
scheme always show higher performance for fixation prediction
than simply training a CNN for each observer. Compared with
MultiConvNets strategy, our multi-task CNN and CNN-PIEF have
the following advantages: i) in both CNN-PIEF and multi-task
CNN, some/all network parameters in our network are actually
shared across different observers. Since the PSM prediction for
different observers are related tasks, and our solutions greatly
reduce the number of parameters to be learnt. Given the limited
number of training samples, our solutions help train a more robust
network, consequently boosts the PSM prediction preformation,
which agrees with existing work for multi-task learning [21] [22],
such multi-task CNN usually boosts the performance. However,
given the limited number of training samples, it is unlikely to train
a robust network for each observer in MultiConvNets; ii) it is also

Methods
RGB based MultiConvNets
RGB based CNN-PIEF
RGB based Multi-task CNN
LDS [25]
LDS based MultiConvNets
LDS based CNN-PIEF
LDS + Multi-task CNN
ML-Net [19]
ML-Net based MultiConvNets
ML-Net based CNN-PIEF
ML-Net + Multi-task CNN
BMS [26]
BMS based MultiConvNets
BMS based CNN-PIEF
BMS + Multi-task CNN
SalNet [10]
SalNet based MultiConvNets
SalNet based CNN-PIEF
SalNet + Multi-task CNN

CC
0.5832
0.6109
0.6013
0.5481
0.5610
0.6532
0.6509
0.3198
0.4310
0.6368
0.6463
0.4937
0.5510
0.6448
0.6390
0.6238
0.6397
0.6771
0.6661

Similarity
0.6425
0.6513
0.7172
0.5777
0.7535
0.7696
0.7792
0.4941
0.7333
0.8095
0.8077
0.6757
0.7636
0.7931
0.7925
0.6847
0.7442
0.7636
0.7547

AUC-Judd
0.8382
0.8483
0.8465
0.8264
0.8103
0.8494
0.8459
0.7117
0.7711
0.8365
0.8414
0.8009
0.8196
0.8486
0.8472
0.8471
0.8448
0.8588
0.8580

NSS
1.3689
1.4763
1.4771
1.3274
1.3767
1.6638
1.6308
0.5664
0.8733
1.5105
1.4960
1.1241
1.2884
1.6004
1.5464
1.5848
1.6924
1.7819
1.7445

TABLE 2: The performance comparison of different methods
under closed-set settings on our PSM dataset.

worth noting that such MultiConvNets based approach is also not
scalable for PSMs prediction in real applications where there are
many observers, because both the collection of training data and

9

0.5
0.4

LDS

ML−Net

BMS

0.8

0.75

0.7

SalNet

LDS

(a) CC (Multi-task CNN)

BMS

0.85

0.8

0.75

SalNet

With Supervision
Without Supervision

LDS

(b) Similarity (Multi-task CNN)

0.84
With Supervision
Without Supervision

0.65

0.6

Metric value

0.7

Metric value

ML−Net

0.9

With Supervision
Without Supervision

0.82
0.8
0.78
0.76

ML−Net

BMS

SalNet

ML−Net

BMS

(e) CC (CNN-PIEF)

SalNet

LDS

ML−Net

BMS

SalNet

(f) Similarity (CNN-PIEF)

1.6
1.4
1.2
1
LDS

(c) AUC-Judd (Multi-task CNN)

With Supervision
Without Supervision

0.86
0.85
0.84
0.83

0.74
LDS

With Supervision
Without Supervision

1.8

0.8

0.82

LDS

ML−Net

BMS

SalNet

Metric value

0.6

With Supervision
Without Supervision

Metric value

0.7

0.85

Metric value

With Supervision
Without Supervision

Metric value

Metric value

0.8

Metric value

Fig. 7: Some examples of PSMs predicted by our methods.

ML−Net

BMS

SalNet

(d) NSS (Multi-task CNN)

With Supervision
Without Supervision

1.8
1.7
1.6
1.5
1.4
1.3

LDS

(g) AUC-Judd (CNN-PIEF)

ML−Net

BMS

SalNet

(h) NSS (CNN-PIEF)

Fig. 8: The effect of supervision on middle layers on our Multi-task CNN and CNN-PIEF.

the phase of training multiple networks are time consuming. In
contrast, our solution, can reduce the time costs in both network
training. Further, as shown in the following section (section 5.3),
for a unseen observer, our CNN-PIEF model can be easily transferred to this new observer and achieves satisfactory performance,
which further validates the scalability of our solutions.
5.3

Methods
LDS [25]
LDS based CNN-PIEF
LDS based CNN-PIEF (transferred model)
ML-Net [19]
ML-Net based CNN-PIEF
ML-Net based CNN-PIEF (transferred model)
BMS [26]
BMS based CNN-PIEF
BMS based CNN-PIEF (transferred model)
SalNet [10]
SalNet based CNN-PIEF
SalNet based CNN-PIEF (transferred model)

CC
0.5765
0.6638
0.6284
0.3317
0.6450
0.6117
0.4637
0.6506
0.6038
0.5981
0.6863
0.6468

Similarity
0.5839
0.7771
0.7626
0.4973
0.8166
0.7946
0.6641
0.7995
0.7818
0.6699
0.7688
0.7519

AUC-Judd
0.8528
0.8693
0.8601
0.7350
0.8559
0.8534
0.8008
0.8685
0.8629
0.8629
0.8811
0.8740

NSS
1.5336
1.8147
1.7239
0.6636
1.6879
1.5490
1.1531
1.7864
1.5978
1.6701
1.9877
1.8484

Performance Evaluation Under Open-Set Setting

To evaluate the generalization capability of our model, we directly
adopt the model trained on closed-set setting and test it with the
600 testing images corresponding to the 10 unseen observers in
the open-set. The performance of different methods are shown in
Tab. 3. The baselines (X based CNN-PIEF) in Tab. 3 are trained
with the 1000 training images corresponding to the 10 observers
in the open-set setting, while the “transferred model” corresponds
to the model trained on 20 observers in the closed-set setting.
Despite being trained with PSMs on totally different observers,
the transferred models could beat USM in terms of all metrics,
while still maintains a comparable performance against models
that are specifically trained for the 10 observers. It’s worth
mentioning that the transferred model is only trained on a small
dataset of 20 observers, further performance boost can be expected
on a larger dataset with more observers. The good performance
of the transferred models in this experiments demonstrate the

TABLE 3: The performance comparison of different methods
under open-set setting on our PSM dataset. The baselines in
this table are trained with 1000 images corresponding to the 10
participants in the open-set setting, while the transferred model
are trained on the training data of our closed-set setting

scalability of our model for PSM prediction, i.e. , once we collect
the PSMs and their person specific information for a large amount
of observers, a model with good generalization capability can be
trained. For a new observer, we can easily collect his/her person
specific information, enabling us to predict the PSM of him/her
over any give image. It takes about 10 hours for each participant
to observe 1K test images 4 times. On the contrary, personspecific information can be easily acquired by answering a few
questions, which can be done in only a few mins. Further, even

10
) ) ) ) ) 0 0 0 0 0
) ,1)         
)  ,1)        
)   ,1)       
)    ,1)      
)     ,1)     
0      ,1)    
0       ,1)   
0        ,1)  
0         ,1) 
0          ,1)

) ) ) ) ) 0 0 0 0 0
) ,1)         
)  ,1)        
)   ,1)       
)    ,1)      
)     ,1)     
0      ,1)    
0       ,1)   
0        ,1)  
0         ,1) 
0          ,1)

(a) CC

(b) Similarity

(e) Female
/ / / / / ' ' ' ' '
/ ,1)         
/  ,1)        
/   ,1)       
/    ,1)      
/     ,1)     
'      ,1)    
'       ,1)   
'        ,1)  
'         ,1) 
'          ,1)

(a) CC

)
)
)
)
)
0
0
0
0
0

)
,1)










)

,1)









/
,1)










/

,1)









/


,1)








/



,1)







/




,1)






'





,1)





'






,1)




)



,1)







)




,1)






0





,1)





0






,1)




0







,1)



0








,1)


0









,1)

) ) ) ) ) 0 0 0 0 0
) ,1)      
  
)  ,1)    
 
 
) 
 ,1) 
     
)    ,1)     

)  
  ,1)
 
  
0    
 ,1)    
0       ,1)   
0        ,1)  
0         ,1) 
0          ,1)

(c) AUC-Judd

(f) Female
/
/
/
/
/
'
'
'
'
'

)


,1)








(d) NSS

(g) Male
'







,1)



'








,1)


'









,1)

(b) Similarity

/
/
/
/
/
'
'
'
'
'

/
,1)










/

,1)









/


,1)








/



,1)







/




,1)






'





,1)





'






,1)




(h) Male
'







,1)



'








,1)


'









,1)

/
/
/
/
/
'
'
'
'
'

/
,1)










/

,1)









(c) AUC-Judd

/


,1)








/



,1)







/




,1)






'





,1)





'






,1)




'







,1)



'








,1)


'









,1)

(d) NSS

(e) Liking Sports

(f) Liking Sports

(g) Disliking Sports

/ / / / / ' ' ' ' '
/ ,1)         
/  ,1)        
/   ,1)       
/    ,1)      
/     ,1)     
'      ,1)    
'       ,1)   
'        ,1)  
'         ,1) 
'          ,1)

/ / / / / ' ' ' ' '
/ ,1)         
/  ,1)        
/   ,1)       
/    ,1)      
/     ,1)     
'      ,1)    
'       ,1)   
'        ,1)  
'         ,1) 
'          ,1)

/ / / / / ' ' ' ' '
/ ,1)         
/
 ,1)
 
     
/

 ,1)
      
/
   ,1)      
/



 ,1)  
  
'    
 ,1) 
  
'       ,1)   
'        ,1)  
'  
 
    ,1) 
' 



     ,1)

(h) Disliking Sports

(a) CC

(b) Similarity

(c) AUC-Judd

(d) NSS

(e) Liking Fashion

(f) Liking Fashion

(g) Disliking Fashion

(h) Disliking Fashion

/
/
/
/
/
'
'
'
'
'

/
,1)










/

,1)









/


,1)








/



,1)







/




,1)






'





,1)





'






,1)




'







,1)



'








,1)


'









,1)

Fig. 9: The CC, Similarity, AUC-Judd, and NSS for participants with the same/different person-specific information. Here ‘F/M-x’
represents the Female/Male with index x, and ‘L/D-x’ represents the xth participant who likes/dislikes sports or fashion, respectively.
Because of space constraint, we only compare the PSMs for 10 participants with the same/different person-specific information.

we collect some training image for a new observer, because of
the limited training samples, it may not contain sufficient personal
information for personalized saliency prediction.

5.4

The effect of supervision on middle layers

Fig. 8 shows the accuracy gain in CC, Similarity, AUC-Judd and
NSS metrics from imposing supervision on middle layers in our
Multi-task CNN and CNN-PIEF based on LDS, ML-Net, BMS,
and SalNet. We observe that middle layer supervision is helpful for
PSM prediction in line with previous findings [20]. By comparing
Fig. 8 with Tab. 2, we can see that even without supervision on
middle layers, our solutions still outperform all baselines, which
further validates the effectiveness of our models.

5.5 The Position of Inserting Person Specific Information
We further empirically investigate how the position of the personspecific information encoded filters would affect the performance
of our network. As shown in Tab. 4, inserting filters in conv5
layer yields the best performance. We also observe a descending
trend for all measurements by inserting the person specific information in higher convolutional layers, which may indicate several
convolutional layers are needed for blending the person-specific
information and image contents for PSMs prediction. Further,
when the filters are inserted in all of the last three layers, the
performance drops even further, possibly due to the dramatically
increased parameters in CNN-PIEF and the unmatched amount of
training samples.

11

Insert Position
Conv5 Conv6 Conv7
Y
/
/
/
Y
/
/
/
Y
Y
Y
Y

CC

Similarity

AUC-Judd

NSS

0.6532
0.6394
0.6163
0.5867

0.7696
0.7709
0.7647
0.7622

0.8494
0.8426
0.8348
0.8229

1.6638
1.6137
1.5290
1.4283

(a)The performance of LDS based CNN-PIEF
Insert Position
Conv5 Conv6 Conv7
Y
/
/
/
Y
/
/
/
Y
Y
Y
Y

CC

Similarity

AUC-Judd

NSS

0.6368
0.6397
0.6222
0.5511

0.8095
0.8082
0.8059
0.7919

0.8365
0.8414
0.8343
0.8013

1.5105
1.4695
1.3987
1.1166

(b)The performance of ML-Net based CNN-PIEF
Insert Position
Conv5 Conv6 Conv7
Y
/
/
/
Y
/
/
/
Y
Y
Y
Y

CC

Similarity

AUC-Judd

NSS

0.6448
0.6292
0.6289
0.5704

0.7931
0.7921
0.7897
0.7721

0.8486
0.8460
0.8457
0.8270

1.6004
1.4857
1.4945
1.3024

(c)The performance of BMS based CNN-PIEF
Insert Position
Conv5 Conv6 Conv7
Y
/
/
/
Y
/
/
/
Y
Y
Y
Y

CC

Similarity

AUC-Judd

NSS

0.6771
0.6298
0.6724
0.6629

0.7636
0.6847
0.7631
0.7563

0.8588
0.8494
0.8583
0.8579

1.7819
1.6157
1.7702
1.7422

(d)The performance of SalNet based CNN-PIEF

TABLE 4: The effect of different inserting positions of personspecific information on X based CNN-PIEF.

5.6 The Effectiveness of Different Person Specific Information
We also shows CC, Similarity, AUC-Judd, and NSS for persons
with similar/dissimilar person-specific information in Fig. 9. We
can see that persons with similar person-specific information
usually corresponds higher consistency in all saliency evaluation
metrics. For example, for images in the first row of Fig. 9, we
can see that persons with the same gender usually have higher
CC, Similarity, AUC-Judd, and NSS. For persons with different
genders, their personalized saliency maps exhibit heterogeneity.
This validates the effectiveness of person-specific information we
used for personalized saliency detection.

6

C ONCLUSION

Motivated by recent psychology studies that saliency is highly
specific than universal, we propose to study the personalized
saliency detection task. We have built the first PSM dataset and the
corresponding person-specific information of 30 subjects. To our
knowledge, this is the first comprehensive study on personalized
saliency and it is expected to stimulate significant future research.
To predict PSM, the wisdom of USM motivates us to decompose PSM as the summation of USM and a discrepancy
between PSM and USM. Then we propose two solutions to
predict such discrepancy: (i) we have presented a Multi-task CNN
framework for the prediction of this discrepancy. (ii) we found
that personalized saliency is closely related to the observers’
personal information (gender, race, major, etc. ). Therefore, as
such discrepancy is image content and identity related, we propose
to concatenate the USM and RGB image and feed them to a
CNN with person-specific information encoded filters to predict
this discrepancy. Extensive experiments validate the effectiveness
of our methods for personalized saliency prediction.

R EFERENCES
[1] L. Itti, ”Automatic foveation for video compression using a neurobiological model of visual attention.” in IEEE Trans. on Image Proc., vol. 13,
no.10, 2004, pp.1304–1318.
[2] V. Setlur, S. Takagi, R. Raskar, M. Gleicher, and B. Gooch, ”Automatic
image retargeting.” in In Proceedings of the 4th international conference
on Mobile and ubiquitous multimedia, 2005, December ,pp. 59–68
[3] M.M.L. Chang, S.K. Ong, A.Y.C. Nee, ”Automatic Information Positioning Scheme in AR-assisted Maintenance Based on Visual Saliency.”
in International Conference on Augmented Reality, Virtual Reality and
Computer Graphics. 2016, pp.453–462.
[4] M. Gygli, H. Grabner, H. Riemenschneider, F. Nater, and L. Van Gool,
”The interestingness of images.” in Proc. IEEE Int. Conf. Comput. Vis.
2013, pp.1633–1640.
[5] A. Borji, M.M. Cheng, H. Jiang, and J. Li, ”Salient object detection: A
survey.” in arXiv preprint arXiv:1411.5878, 2014.
[6] T. Judd, K. Ehinger, F. Durand, and A. Torralba, ”Learning to predict
where humans look.” in Proc. IEEE Int. Conf. Comput. Vis., 2009,
pp.2106–2113.
[7] Y. Li, X. Hou, C. Koch, J.M. Rehg, and A.L. Yuille, ”The secrets of salient
object segmentation.” in Proc. IEEE Conf. Comput. Vis. Pattern Recog.,
2014, pp.280–287.
[8] P. Xu, K. A. Ehinger, Y. Zhang, A. Finkelstein, A. R. Kulkarni, and J. Xiao,
”Turkergaze: Crowdsourcing saliency with webcam based eye tracking.”
in arXiv preprint arXiv:1504.06755, 2015.
[9] X. Huang, C. Shen, X. Boix, and Q. Zhao, ”Salicon: Reducing the
semantic gap in saliency prediction by adapting deep neural networks.”
in Proc. IEEE Int. Conf. Comput. Vis., 2015, pp.262–270.(2015).
[10] J. Pan, E. Sayrol, X. Giro-i-Nieto, K. McGuinness, and N. E. O’Connor,
”Shallow and deep convolutional networks for saliency prediction.” in
Proc. IEEE Conf. Comput. Vis. Pattern Recog., 2016, pp. 598–606.
[11] N. Liu, J. Han, D. Zhang, S. Wen, and T. Liu, ”Predicting eye fixations
using convolutional neural networks.” in Proc. IEEE Conf. Comput. Vis.
Pattern Recog., 2015, pp.362–370.
[12] S. S. Kruthiventi, K. Ayush, and R. V. Babu, ”Deepfix: A fully convolutional neural network for predicting human eye fixations.” in IEEE Trans.
on Image Proc., 2017.
[13] S. S. Kruthiventi, V. Gudisa, J. H. Dholakiya, and R. Venkatesh Babu,
”Saliency unified: A deep architecture for simultaneous eye fixation
prediction and salient object segmentation.” in Proc. IEEE Conf. Comput.
Vis. Pattern Recog., 2016, pp. 5781–5790.
[14] J. Xu, M. Jiang, S. Wang, M. S. Kankanhalli, and Q. Zhao, ”Predicting
human gaze beyond pixels.” in Journal of vision, vol. 14, no.1, 2014,
pp.28–28.
[15] M. Jiang, S. Huang, J. Duan, and Q. Zhao, ”Salicon: Saliency in context.”
in Proc. IEEE Conf. Comput. Vis. Pattern Recog., 2015, pp.1072–1080.
[16] O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, and A.
C. Berg, ”Imagenet large scale visual recognition challenge.” in IEEE Int.
Journal of Compu. Vis., vol. 115, no. 3, 2015, pp.211–252.
[17] B. C. Russell, A. Torralba, K. P. Murphy, and W. T. Freeman, ”LabelMe:
a database and web-based tool for image annotation.” in IEEE Int. Journal
of Compu. Vis., vol. 77, no. 1, 2008, pp.157–173.
[18] T. Judd, F. Durand, and A. Torralba, ”A benchmark of computational
models of saliency to predict human fixations.” in 2012.
[19] M. Cornia, L. Baraldi, G. Serra, and R. Cucchiara, ”A deep multi-level
network for saliency prediction.” in IEEE In Pattern Recognition (ICPR),
2016 23rd International Conference, 2016, pp.3488–3493).
[20] C. Y. Lee, S. Xie, P. Gallagher, Z. Zhang, and Z. Tu, ”Deeply-supervised
nets.” in Artificial Intelligence and Statistics, 2015, pp.562–570.
[21] X. Li, L. Zhao, L. Wei, M. H. Yang, F. Wu, Y. Zhuang, and J. Wang,
”DeepSaliency: Multi-task deep neural network model for salient object
detection.” in IEEE Trans. on Image Proc., vol. 25, no. 8, 2016, pp.3919–
3930.
[22] Z. Zhang, P. Luo, C. C. Loy, and X. Tang, ”Facial landmark detection by
deep multi-task learning.” in Proc. IEEE Eur. Conf. Comput. Vis., 2014,
pp.94–108.
[23] Y. Jia, E. Shelhamer, J. Donahue, S. Karayev, J. Long, R. Girshick, S.
Guadarrama, and T. Darrell, ” Caffe: Convolutional Architecture for Fast
Feature Embedding.” in arXiv preprint arXiv:1408.5093, 2014.
[24] Z. Bylinskii, T. Judd, A. Borji, L. Itti, F. Durand, A. Oliva, and A.
Torralba, ”Mit saliency benchmark.” 2015
[25] S. Fang, J. Li, Y. Tian, T. Huang, and X. Chen, ”Learning discriminative
subspaces on random contrasts for image saliency analysis.” in IEEE
transactions on neural networks and learning systems, vol. 28, no. 5,
2017, pp.1095–1108.
[26] J. Zhang, S. Sclaroff, ”Saliency detection: A boolean map approach.” in
Proc. IEEE Int. Conf. Comput. Vis., 2013, pp.153–160.

12

[27] N. Liu, J. Han, D. Zhang, S. Wen, and T. Liu, ”Predicting eye fixations
using convolutional neural networks.” in Proc. IEEE Conf. Comput. Vis.
Pattern Recog., 2015, pp.362–370.
[28] J. Hewig, R. H. Trippe, H. Hecht, T. Straube, and W. H. Miltner, ”Gender
differences for specific body regions when looking at men and women.”
in Journal of Nonverbal Behavior, vol. 32, no. 2, 2008, pp.67–78.
[29] R. D. Wilson, N. K. Moore, ”The role of sexually-oriented stimuli
in advertising: Theory and literature review.” in ACR North American
Advances, 1979.
[30] N. Alwall, D. Johansson, and S. Hansen, ”The gender difference in gazecueing: Associations with empathizing and systemizing.” in Personality
and Individual Differences, vol. 49, no. 7, 2010, pp.729–732.
[31] J. Carreira, P. Agrawal, K. Fragkiadaki, and J. Malik, ”Human pose
estimation with iterative error feedback.” in Proc. IEEE Conf. Comput.
Vis. Pattern Recog., 2016, pp.4733-4742.
[32] R. J. Dolan, and T. Sharot, ”Neuroscience of preference and choice:
cognitive and neural mechanisms.” in Academic Press., 2011.
[33] J. R. Absher, and J. Cloutier, ”Neuroimaging Personality, Social Cognition, and Character.” in Academic Press., 2016.
[34] J. Hewig, R. H. Trippe, H. Hecht, T. Straube, and W. H. Miltner, ”Gender
differences for specific body regions when looking at men and women.”
in Journal of Nonverbal Behavior, vol. 32, no. 2, 2008, pp.67–78.
[35] M. Imafuku, M. Kawai, F. Niwa, Y. Shinya, M. Inagawa, and M.
Myowa Yamakoshi, ”Preference for Dynamic Human Images and Gaze
- Following Abilities in Preterm Infants at 6 and 12 Months of Age: An
Eye-Tracking Study.” in Infancy, vol. 22, no. 2, 2017, pp.223–239.
[36] ’Gender Specific Gaze Differences?’, in MRC International Report,
2012.
[37] H. Noh, P. H. Seo, and B. Han, ”Image question answering using
convolutional neural network with dynamic parameter prediction.” in
CoRR, 2015, pp.30–38.
[38] B. De Brabandere, X. Jia, T. Tuytelaars, and L. Van Gool, ”Dynamic
filter networks.” in Proc. Adv. Neural Inf. Process. Syst., 2016.
[39] C. Finn, I. Goodfellow, and S. Levine, ”Unsupervised learning for
physical interaction through video prediction.” in Proc. Adv. Neural Inf.
Process. Syst., 2016, pp.64–77.
[40] A. Krizhevsky, I. Sutskever, and G. E. Hinton, ”Imagenet classification
with deep convolutional neural networks.” in Proc. Adv. Neural Inf.
Process. Syst., 2012, pp.1097–1105.
[41] J. Long, E. Shelhamer, and T. Darrell, ”Fully convolutional networks for
semantic segmentation.” in Proc. IEEE Conf. Comput. Vis. Pattern Recog.,
2015, pp.3431–3440.
[42] E. Vig, M. Dorr, D. Cox, ”Large-scale optimization of hierarchical
features for saliency prediction in natural images.” in Proc. IEEE Conf.
Comput. Vis. Pattern Recog., 2014, pp.2798–2805.
[43] M. Kmmerer, L. Theis, and M. Bethge, ”Deep gaze i: Boosting saliency
prediction with feature maps trained on imagenet.” in arXiv preprint
arXiv:1411.1045, 2014.
[44] K. Simonyan, and A. Zisserman, ”Very deep convolutional networks for
large-scale image recognition.” in arXiv preprint arXiv:1409.1556, 2014.
[45] C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, and
A. Rabinovich, ”Going deeper with convolutions.” in Proc. IEEE Conf.
Comput. Vis. Pattern Recog., 2015, pp.1–9.
[46] L. Itti, C. Koch, and E. Niebur, ”A model of saliency-based visual
attention for rapid scene analysis.” in Proc. Int. Conf. Mach. Learn., 1998,
vol. 20, no. 11, pp. 1254–1259.
[47] N. Riche, M. Mancas, M. Duvinage, M. Mibulumukini, B. Gosselin,
and T. Dutoit. ”Rare2012: a multi-scale rarity-based saliency detection
with its comparative statistical analysis.” in Signal Processing Image
Communication, 2013, vol. 28, no. 6, pp. 642–658.
[48] E. Erdem, and A. Erdem, ”Visual saliency estimation by nonlinearly
integrating features using region covariances.” in Journal of Vision, 2013,
vol. 13, no. 4, 11.
[49] A. Borji, ”Boosting bottom-up and top-down visual features for saliency
estimation.” in Proc. IEEE Conf. Comput. Vis. Pattern Recog., 2012, pp.
438–445).
[50] T. Judd, K. Ehinger, F. Durand, and A. Torralba, ”Learning to predict
where humans look.” in Proc. IEEE Int. Conf. Comput. Vis., 2010, pp.
2106–2113).
[51] Y. Xu, N. Li, J. Wu, J. Yu, Sh. Gao, ”Beyond Universal Saliency:
Personalized Saliency Prediction with Multi-task CNN”, in Proceedings of
the Twenty-Sixth International Joint Conference on Artificial Intelligence,
2017, pp. 3887–3893.
[52] P. Xu, K. A. Ehinger, Y. Zhang, A. Finkelstein, S. R. Kulkarni, and
J. Xiao, ”Turkergaze: crowdsourcing saliency with webcam based eye
tracking.” in Computer Science., 2015.

Yanyu Xu received the bachelor degree from
Dalian University of Technology, Dalian, China.
He is currently pursuing Ph.D. degree at the
ShanghaiTech University.

Junru Wu received the bachelor degree from
Tongji University, Shanghai, China. He is currently pursuing Ph.D. degree at the ShanghaiTech University.

Nianyi Li received the BE degree from the Department of Electrical Engineering, Huazhong
University of Science and Technology, in 2013.
She is now working toward the PhD degree
in the Department of Computer and Information Sciences, University of Delaware. Her research interests include saliency detection, object recognition, super resolution, scene reconstruction, light field rendering, and non-pinhole
camera models. She is a student member of the
IEEE.

Shenghua Gao is an assistant professor in
ShanghaiTech University, China. He received
the B.Eng. degree from the University of Science
and Technology of China in 2008 (outstanding
graduates), and received the Ph.D. degree from
the Nanyang Technological University in 2012.
From Jun 2012 to Jul 2014, he worked as a
postdoctoral fellow in Advanced Digital Sciences
Center, Singapore. His research interests include computer vision and machine learning.

Jingyi Yu received the BS degree from Caltech,
in 2000, and the MS and PhD degrees in EECS
from MIT, in 2005. He is a professor in the Computer and Information Science Department, University of Delaware. His research interests span
a range of topics in computer graphics, computer
vision, and image processing, including computational photography, medical imaging, nonconventional optics and camera design, tracking and
surveillance, and graphics hardware. He is a
senior member of the IEEE.

