Where’s YOUR focus: Personalized Attention
Sikun LIN

Pan Hui

Hong Kong University of Science and Technology
Hong Kong
sklin@connect.ust.hk

Hong Kong University of Science and Technology
Hong Kong
panhui@cse.ust.hk

arXiv:1802.07931v1 [] 22 Feb 2018

ABSTRACT
Human visual attention is subjective and biased according to the personal preference of the viewer, however, current works of saliency
detection are general and objective, without counting the factor of
the observer. This will make the attention prediction for a particular
person not accurate enough. In this work, we present the novel
idea of personalized attention prediction and develop Personalized
Attention Network (PANet), a convolutional network that predicts
saliency in images with personal preference. The model consists of
two streams which share common feature extraction layers, and
one stream is responsible for saliency prediction, while the other
is adapted from the detection model and used to fit user preference. We automatically collect user preference from their albums
and leaves them freedom to define what and how many categories
their preferences are divided into. To train PANet, we dynamically
generate ground truth saliency maps upon existing detection labels and saliency labels, and the generation parameters are based
upon our collected datasets consists of 1k images. We evaluate the
model with saliency prediction metrics and test the trained model
on different preference vectors. The results have shown that our
system is much better than general models in personalized saliency
prediction and is efficient to use for different preferences.

CCS CONCEPTS
• Computing methodologies → Interest point and salient
region detections; • Human-centered computing → Human
computer interaction (HCI);

KEYWORDS
Saliency Prediction, Personalized HCI, Deep Network
ACM Reference Format:
Sikun LIN and Pan Hui. 2018. Where’s YOUR focus: Personalized Attention.
In Proceedings of (N/A). ACM, New York, NY, USA, 9 pages. https://doi.org/
10.475/123_4

1

INTRODUCTION

Attention is a personalized experience, different people may focus
on different regions even they are facing at a same scene: in a park
containing both entertainment equipment and children playing
around, attention of parents may be focused on children, whereas
children may focus on the equipment. Predicting correctly where
Permission to make digital or hard copies of part or all of this work for personal or
classroom use is granted without fee provided that copies are not made or distributed
for profit or commercial advantage and that copies bear this notice and the full citation
on the first page. Copyrights for third-party components of this work must be honored.
For all other uses, contact the owner/author(s).
N/A, 2017,
© 2017 Copyright held by the owner/author(s).
ACM ISBN 123-4567-24-567/08/06.
https://doi.org/10.475/123_4

the attention is for each user is crucial for a Human-ComputerInteraction (HCI) application. With recent advances in deep learning and the improvement of computation powers, vision tasks such
as object detection and saliency prediction achieve higher accuracy
and faster implementations. It is feasible now to create a model for
predicting user attention that can fit user preferences differently.
Preference here means the different tendencies to focus on various
objects, which is inherently consistent for one particular user.
Such a model can be a part of an augmented reality (AR) recommendation system that retrieves and displays only information that
the user is in favour of, focusing the post processing only on useful
regions. It is also helpful in achieving customized scene description
or video summarization for individual users. The state-of-the-art
saliency prediction works are all trained and evaluated on objective
datasets [4, 7, 17]. To fulfill the increasing demand of providing
more fitting user experience, we propose in this work, Personalized
Attention Network (PANet), a deep architecture that can predict
personalized saliency areas.
PANet consists of two streams of convolutional neural networks
(CNNs) that share common feature extraction layers. The model
takes three inputs: raw image to be processed, user-defined detailed
class to super category mapping, and user preference vector on
super categories. Given the input image, PANet will extract its
deep features at multiple scales and pass them to two streams: the
saliency prediction stream will generate a saliency map without the
influence of user preference, and the preference fitting stream will
utilize object detection model architecture to generate a preference
map according to the input preference. After combining the results
gotten from two streams, post-processing including adding a centrer
prior will be done and the prediction result will be given as a
pixel-level saliency map that fits this particular user. To train a
PANet model, we regress pixel-wisely on the ground truth, which
is generated dynamically in the training generator given the input
preference.

2

RELATED WORKS

Although state-of-the-art saliency prediction models perform well
in terms of evaluation metrics, they are all measured on benchmark
datasets considering no personal preference factor. On the other
hand, a subjective version of saliency prediction is needed in the
real world for HCI applications. In our work, we combine object
detection and saliency prediction techniques to build a model for
predicting personalized attention.

2.1

DNNs for object detection

Traditional detection pipeline typically consists of multiple steps:
feature [2, 29, 35] detection and descriptor extraction [1, 5] , feature
representation [8, 31], and classifiers [3, 43] applied in a sliding
window manner [10] or in selected image sub-regions [42]. After

N/A, 2017,
CNN gains its popularity, [40] tried to solve detection problem in a
regression manner, however it only achieves 30.5% mAP on PASCAL
VOC2007 [9]. As CNN is good at classification tasks, R-CNN [12]
tries to combine region proposal and CNN-based classification for
detection, whose good performance makes region proposal deep
detection systems popular. SPP-net [14] solves the issue of repeated
calculation and speed up the whole process by putting the feature
extraction convolutional layers at the beginning and introducing
spatial-pyramid pooling layer, making later classification layers
share the features. Fast R-CNN [11] then refines ROI pooling layer
that generates potential bounding boxes, further making the whole
model end-to-end trainable. Faster R-CNN [34] tackles the speed
bottleneck, selective search, by proposing Region Proposal Network
(RPN), letting the network itself learn to generate region proposals.
Even faster implementations are achieved by omitting region
proposals completely and segment image into default grid cells
and anchor boxes, modeling detection as regression at each cell
or anchor box, and simultaneously learn multiple targets: object
class, bounding box position, and prediction confidence. YOLO [32]
segments input images into 7 × 7 grid cells, each responsible for
predicting 2 bounding boxes and one object class.This results in a
faster speed, but at the same time imposes the limitation to detect
small or overlapped objects. SSD [28] performs better as it generates
default anchor boxes in different aspect ratios at each cell and at
multiple feature scales. Class prediction is not bound with cells but
with anchor boxes, making it more flexible in predicting multiple
objects within a small sub-region. It also replaces fully connected
prediction layers in YOLO with convolutional ones, further increasing the speed. To solve the detection problem with small objects,
YOLO v2 [33] fine-tunes the classifier network with higher input
resolution to make high-resolution detection part adjust better. It
also removes the fully connected layers for bounding box prediction
and use anchor boxes as in SSD. Its customized network structure
and carefully designed training techniques altogether make a good
end performance. Part of our model utilize the network structure
of SSD, serving as part of the preference fitting stream, as well as
the provider of multi-scale feature maps for the saliency prediction
stream.

2.2

DNNs for Saliency Prediction

Saliency prediction can be categorized into three general categories:
bottom-up approaches based on low-level features such as color,
contrastness, texture, etc. [7, 18, 30, 45–47]; top-down approaches
based on high-level image features such as object knowledge [13,
16, 38]; and the combination of the two [6, 44]. Development in
deep learning boosts the performance of saliency prediction, where
saliency datasets used for training and benchmark (SALICON [17],
MIT300 [4], etc.) also make important contributions.
Recent works using deep networks in saliency detection obtain
good performance, as the networks can extract more robust features than handcrafted low-level features. [15, 26] convert the input
image into different resolutions by down-sampling, and feed them
into multiple CNN streams to extract multi-scale features, and then
pass the combined feature to post-processing layers to get the final saliency map. [20] directly uses truncated VGG-16 [39] layers
for feature extraction, and add Inception modules to capture the

Sikun LIN and Pan Hui
multi-scale information. [21–23] use only one CNN stream, but
get features in different scales through different layers, and rescale
all these feature maps to a same size for further feature merging
and saliency map generation. For the saliency detection part in
our work, we use the deep features extracted from VGG-16 trained
on ImageNet [36]. Meanwhile, the feature maps at different scales
extracted from detection layers are merged in saliency prediction
stream for further processing.
Several works [19, 22, 27] include center prior as human generally focus their attention on the center of their eyesights. The
priors are added either as a Gaussian function of the pixel distance
to the center, or gotten from labeled saliency datasets. Adding center prior may have a negative influence on the result depending
on the task, such as predicting eye fixations on a webpage, where
human may tend to look for information on the edge menus. Our
work considers only general scenes, thus center prior is added and
can make positive contribution to the final result.

3

PREFERENCE IDENTIFICATION

Objects can be categorized into both detailed and super categories:
for example, an apple belongs to category “apple” and super category “fruit”. MS COCO objects have both labels, thus infers a default
mapping between categories and super categories. However, this
mapping has disadvantages: some super categories contain classes
that are likely to co-occur, such as objects belong to “appliance” and
“kitchen”, and their preference scores prone to be similar to each
other for a fixed user. A more meaningful mapping should merge
the super categories that tend to occur in a same environment or
have a similar usage. In addition, this default mapping may not
fit individual needs: one user may want a mapping points to “outdoor” and “in-door”, whereas another user may want “human” and
“non-human” super categories to represent his preference. These
limitations make a general default mapping not suitable for a personalized system, so we leave the freedom of defining the mapping
to user themselves.
To collect the preference of a particular user, we pass images
in user’s smart device into a trained detection model, in our case
the SSD model. We only select image files that have last modified
dates within three months to get current preference of the user. By
counting the occurrence of objects falling into different categories
and taking into account the prediction confidence, we can roughly
know the user preference. The preference towards a particular
super category SCati is defined as:
Õ
Pref i =
Conf x ,
x ∈SCat i

in which x is the object belongs to this category, and Conf x is the
confidence of that prediction. In this manner, more occurrence of
the objects in category SCati will lead to a higher preference of it,
and detection with low confidence will not influence the preference
much. After iterating over all the images, the preference vector can
be obtained by normalizing the preference values to between 0 and
1:
Pref i
Pref i =
,
max Pref j
j

Where’s YOUR focus: Personalized Attention

N/A, 2017,

Figure 1: PANet architecture. The model takes three inputs: original image, personal preference vector pvec, and detailed
class to super category mapping. The upper stream is responsible for saliency detection, and the lower stream is responsible
for fitting user preference. Final output is personalized saliency prediction of the input image.
then we can get the final preference vector of the user: pvec =
[Pref 1 , ..., Pref n ], where n is the total number of super categories.
This approach may suffer from accuracy problem when trained
detection model covers not enough categories. Thus we offer users
the chance to rate preference by themselves: we first calculate pvec
and present it to the user. If user thinks this pvec cannot truly reflect
his preference, he can then rate each super category respectively.

4

MODEL ARCHITECTURE

As shown in Figure 1, PANet contains two streams: the upper one
is for predicting the general saliency information, and the lower
one is for fitting user preference upon the information gotten from
detection layers. They share common image features that extracted
by VGG-16 without its final classification layers as well as the
convolutional layers in the detection part.

4.1

Saliency prediction stream

Saliency prediction part is the upper stream in Figure 1. In order
to combine multi-scale features of the input image for saliency
prediction, but without an extra feature extraction stream as in the
[15, 24], our model uses the features extracted in different layers
from VGG-16 and SSD customized layers. Features at three different
scales are used: conv4_3, conv5_3, conv6_2 (layer names consistent
with those in [37]), with size 38 × 38, 19 × 19, 10 × 10 respectively.
Latter two feature maps are up-sampled to the same size as the
first one. In our implementation, we find that combining features
from the second and third scales can notably improve saliency
prediction accuracy. After rescaling, feature maps are combined as
a 3-dimensional tensor, with size 38 × 38 × 3 and 512 channels. The
combined tensor is then passed through four 3-dimensional kernel 1
convolutional layers, with 64, 128, 4, 1 feature channels respectively.
Then the network reshapes the tensor back to 2 dimension with
size 38 × 38 and 3 channels, and passes it through one more 1 × 1
convolutional layer, from which the network outputs the final result
with a single feature channel for the saliency detection stream.

Figure 2: NMS layer: Tensor creation from kept predictions.

4.2

Preference fitting stream

This stream starts with detection layers labeled as SSD ADD-ONs
in Figure 1, which is the same as those customized layers in SSD
model, containing its featured anchor box generation layers and
produces feature maps at multiple scales. The output of this part
is a concatenation of object category, confidence and coordinate
information, which needs to be converted back to a image-shaped
tensor in our customized NMS and Mapping layers for further
processing.
Non-Maximum Suppression (NMS) operation is performed on
above output to collapse overlapped predictions to one. In NMS
layer, network also choose whether to keep a detection based on its
confidence. The value of confidence threshold varies for different
datasets. For PASCAL VOC2007, threshold 0.6 is enough to get
most of the true positives kept, and eliminate most false positives.
However, for MS COCO dataset that contains many small objects,
the threshold needs to have a smaller value so as to get a satisfactory
recall: we set it to 0.5, which can detect most of the small objects
and have a reasonable false positive rate.
NMS layer then translate the information representing those
predictions with high confidence into a 2-dimensional tensor in
image space, as shown in Figure 2. To merge with the saliency
prediction stream later, the created tensor served as the output of
NMS layer is set to have a size of 38 × 38, and its channel number
N is the same as the number of detailed classes. Each channel
represents the prediction of one particular class. For an input image,
if there exists predictions of objects in category Cati , then the i th

N/A, 2017,

Sikun LIN and Pan Hui

Figure 3: Mapping Layer: Combine multiple channels given
mapping between super categories to categories.

Figure 5: Matlab GUI for collecting user labels.
it to [0, 1]:
Figure 4: Mapping Layer: An example of a combined channel
from two channels.

channel of the tensor will have non-zero pixels according to the
predicted position and prediction confidence. The value at each
pixel (x, y) is max(Con f 1 , · · · , Con fk ), where Con f 1 , · · · , Con fk are
the confidence of predictions that have bounding boxes enclosing
pixel (x, y).
The output channels of NMS layer are in detailed categories,
while Mapping layer combines them into channels representing
super categories. Mapping layer takes in two extra inputs: user
preference vector and defined mapping between super categories
to categories. Given such a mapping:
{SCat 1 → Cat 11 , · · · , Cat 1k1 ;
..
.
SCatn → Catn1 , · · · , Catnkn } ,
tensor channels representing Cati j (∀j) will be merged into a single
channel SCati : Figure 3 shows for an particular mapping, how multiple channels representing different categories are combined to channels that represent super categories. The pixel-wise value of the new
channel representing SCati is: SCati = max(Cati j ×pvec[SCatIdi ]),
j

where pvec[SCatIdi ] represents the preference towards category
SCati . The process of this mapping operation is summarized as in
Figure 4, and it is the key layer that makes the model take in and
fit user preference.

4.3

prior =

Õ

SALдt ,

prior − min(prior )
max(prior ) − min(prior )
Finally, a softmax activation layer is added to output the final prediction as a probability map.
prior =

5

DATA

Currently there is no personalized attention annotation available.
To train our model, we need ground truth data labeled with consistent preference. In our implementation, we first collect a small
dataset labeled by two subjects with their own preferences, and
then we generate more ground truth data according to our collected
labels.

5.1

Data Collection

To collect this dataset, we asked two subjects A and B to firstly
define their preference super categories, then rate each category in
the scale of 0 to 10, the higher rating means the subjects is more
interested in that category. We re-scale their rating to 0 to 1 to get
their corresponding pvec, listed in Table 1. After this, we select
a subset of images from SALICON dataset which consists of 15k
images. The selected images contain at least one object that belongs
to the categories rated larger than 5, as labeling images having both
preferred objects and other objects can make the label more useful.
This turned out to be 3126 images for subject A and images 3974
for subject B: we randomly choose 500 images for each subjects to
label.
A
[cat, car, others] = [1.0, 0.8, 0.2]
B [keyboard, f ood, person, others] = [1.0, 0.8, 0.5, 0.3]
Table 1: Subject preference vectors in our collected dataset.

Merging two streams

The model merges two streams together by tensor concatenation,
and two 1 × 1 convolutional layers with channel number 8 and 1 are
added for more nonlinearity. Further more, as people tend to focus
on the central part of their eyesight, we add a center prior to our
model before the final activation layer, and we generate this prior
map from saliency labels in SALICON dataset by summing up all
the saliency ground truth SALдt in the dataset, and then normalize

The presentation of image is done in Matlab R2015b, and has a
GUI as shown in figure 5, in which the mouse cursor is represented
as a red circle. Label collection is done through mouse tracking
and has a similar procedure as [17]. Each image is presented to
the subject and the subject is asked to move the mouse cursor to
where they want to look at. A mouse click will trigger the recording

Where’s YOUR focus: Personalized Attention

N/A, 2017,

Figure 6: COCO image, visualized ground truth label (subject A), our generated ground truth.
SALдt in SALICON dataset as prior, redistributing the attention
according to pMap. In addition, SALдt is added to make sure salient
area receives fair amount of attention even it is not preferred by the
user. There are also regions containing objects in the preferred categories, but have low saliency probability in SALICON annotations:
to give a chance to these areas, pMap itself is also added:
PSALдt = α SALдt + β SALдt · pMap + γ pMap ,
(a) Sum of CC and SIM at differ- (b) Sum of CC and SIM at different α , with fixed β : γ = 0.8 : 0.2 ent β : γ , with fixed α = 0.06. x
coordinates is the bet a ratio.

Figure 7: Choosing weights for ground truth generation.

process as well as finishing current recording. When the subject
finishes current image annotation, he will need to hit Next button
for the next image to show up. There will be 2 seconds waiting
interval showing a blank figure between two images. As in [17], 50
images form a block, and subjects are allowed to take a short break
between blocks. To make our result stable, we ask each subject
annotation their 500 images five times, in five separate days. The
final fixation annotation for each image is averaged from those five
annotations. The middle column of Figure 6 shows the visualized
results.

5.2

Data Generation

Taking into account that preference is unique, and it is impractical
to build a dataset with preferences cover all typical users, it is
better to use ground truth generated dynamically based on input
preference. This can make us train PANet with more data, and make
the training flexible on new preferences, so that we do not need to
ask each new user using our model to label thousands of images
with their preferences. We generate our training ground truth data
by utilizing currently available detection and fixation annotations,
in our case the MS COCO dataset [25] and its attention annotation
dataset SALICON [17].
Given a particular pvec, we first generate a preference map pMap
using the ground truth bounding box positions and object category information: pMap[x, y] = max(pvec[SCatId[x, y]]) , where
SCatId[x, y] are the super category IDs of the objects cover pixel
[x, y], and we select the maximum preference when multiple objects
co-occur at this position. Then we use the saliency ground truth

where PSALдt is the generated personal saliency ground truth with
the preference pvec. To choose values of the weights, we first fix
β : γ ratio to be 0.8 : 0.2, and find α = 0.06 will give the best
performance, as shown in Figure 7a, in terms of average CC and
SIM scores (metrics explained in Section 7.1). After determining the
most fitting α, we change the ratio between β and γ , the generation
performance is shown in Figure 7b, with β : γ = 0.8 : 0.2 getting
the best score. Under the constraint α + β + γ = 1, the final wights
are 0.06, 0.752 and 0.188 respectively.
Generated ground truth are then normalized and saved as a
probability distribution by going through a softmax:
PSALдt =

PSALдt − min(PSALдt )
,
max(PSALдt ) − min(PSALдt )

e xi
xi = Í x j ,
je
where x i denotes for every pixel value in each generated PSALдt .
We show the comparison of labeled ground truth and our generated
one in Figure 6.

6

TRAINING

We train PANet in three phases. Input images fed to the whole
model are rescaled to 300 × 300, and augmented through crop, flip,
and color jitter, each with probability 0.5. For all training phases, we
keep a track of the minimum validation loss, and if the validation
loss of three continuous epochs are all larger than the minimum
one, we assume the model starts to overfit and stop training.
The first phase is pretraining a SSD model. PANet will use the
weights of SSD300 layers. To get the weights trained on our desired
data, we pretrain a SSD300 model on MS COCO dateset which
contains 80k training images and 40k validation images. Its feature
extraction layers, namely VGG-16 without the final classification
layers, have the weights pretrained on ImageNet. The training
process is adapted from [28, 37]. According to MS COCO section in
[28], the scale of the default boxes are set to be smaller than normal

N/A, 2017,
to fit small objects in this dataset. As for the default box aspect
ratio, we don’t use the generation method as in [28], instead, we
first get a prior box distribution from ground truth bounding boxes,
and then use it for the default box generation. We train the model
for 180k iterations with batch size 20, and the learning rate starting
from 3 × 10−4 , decreasing every epoch with a rate of 0.9.
The second phase is pretraining saliency layers. Without merging
the result from the preference fitting stream, the weights of those
layers responsible for saliency prediction (labeled red in Figure 1)
are pretrained. This will improve model performance compared
with directly training the entire model. We import and freeze the
weights from previous phase into this phase in order to extract
image features. The model is trained on the original SALICON
dataset, which contains 10k training images and 5k validation images. The ground truth label is unbiased saliency annotation. We
directly regress the predicted saliency in probability distribution to
the ground truth distribution pixel-wisely. Training objective for
this phase is KL-divergence:
Õ
pi
D K L (p|q) =
pi log
q
i
i
which is a common metric for saliency prediction models. We
trained this part for 30k iterations with batch size 20, and the learning rate starts from 10−3 , decreasing by 0.9 per epoch.
The final phase is training entire PANet on our dynamically
generated ground truth containing 10k training images and 5k
validation images. For our training, randomly generated preference
vector and default MS COCO category mapping are served as extra
inputs to the model. We load and freeze the weights of VGG part and
SSD add-on layers, while the weights learned for saliency prediction
layers are loaded and fine-tuned. Layers without pretrained weights
are initialized with random weights. The training objective is again
KL-divergence, and we train the model for 50k iterations with batch
size 20 and the same learning rate as in the previous phase.

7

EVALUATION

In this section, we will first describe our results in the quantitative
manner, and then show its prediction results qualitatively. As we
do not want to retrain our model every time when it is applied
to a new preference, we will present the experimental results of
using a trained model to a new preference vector and user-defined
mapping in Section 7.3. We will also compare our model with a
objective saliency model.

7.1

Quantitative Results

MS COCO images, thus its subset SALICON, generally contains
multiple objects and covers different super categories in one image,
making it suitable for our experiment. We randomly re-split the 15K
images in SALICON into 7K training images, 3k validation images
and 5k test images, and then train the model with preference vector
shown in Figure 8. We compare our model with two baselines. The
first is center prior, which sets the saliency baseline. The second is
detection baseline: for each image, we highlight detected objects
with detection confidence × object preference. One problem of the
second baseline is that there exist images that no object is detected
with confidence higher than the 0.5. This will result in zero matrices
which cannot be used in metrics. To solve this, we replace these zero

Sikun LIN and Pan Hui
matrices with randomly generated matrices that are normalized
to have a pixel-sum 1. In the following, we use p,q to represent
predicted saliency map and dynamically generated ground truth
respectively.
We calculate Pearson’s Correlation Coefficient:
σ (p, q)
CC(p, q) =
σ (p) × σ (q)
where σ (p, q) is the covariance of p and q. The average CC score
is 0.725. The center prior baseline has an average CC 0.420, and
detection baseline has a average CC 0.493. These indicate that PANet
predicts much better than the two baselines.
Apart from the linear relationship, we also measure the Similarity
between predicted saliency map and our ground truth:
Õ
SIM(p, q) =
min(p, q) ,
i

Í
Í
where i pi = i qi = 1. Two same saliency maps will have
SIM(p, q) = 1. Different from CC that is calculated symmetrically
and measures false positive and false negative equally, SIM metric
penalizes false positives less than false negatives, making the center
prior baseline scores higher under this metric. The average score
on the test set is 0.742 and the center prior baseline has SIM value
0.622 comparing with the ground truth, while the detection baseline
scores 0.587.
Kullback-Leibler divergence measures the difference between
two saliency maps that are viewed as distributions, and we used
KL-Judd variant in our measurement:
Õ
qi
KL(p, q) =
qi log(ϵ +
),
ϵ
+
pi
i
where ϵ = 2.2204e − 16 is a regularization constant. KL-divergence
is non-symmetric, and penalizes the pixel predictions where qi
are much larger than pi . For our experiment on MS COCO with
this particular preference vector, the average KLD is 0.159, and the
center prior baseline is 0.316. Detection baseline has a much higher
averge KLD score 11.222 as its distribution is much different from
a saliency map. We will also see in Section 7.3 that the variance of
preference vector will change KLD score significantly, making it
less suitable for measuring our model performance.
Earth Mover’s Distance (EMD) measures the spatial distance
between distributions by computing the minimum cost of moving
densities from one distribution into another. The version we use in
our experiment is its linear variant as in [4]:
Õ
Õ
Õ
EMD(p, q) = min
fi j di j + |
pi −
q j | max di j ,
{f i j }

i, j

i

j

ij

under constraints: (1)fi j ≥ 0, (2) j fi j ≤ pi , (3) i fi j ≤ q j ,
Í
Í
Í
(4) i, j fi j = min( i pi , j q j ), where di j is the ground distance
between i th and j th bin of the distributions, and fi j is the density
flow between them. The average EMD score on the test set for this
particular preference vector is 0.685, with center prior scoring 3.668
and detection baseline scoring 5.116.
Í

7.2

Í

Qualitative Results

We show our model predictions in Figure 8, which are tested upon
MS COCO images with the preference vector listed in the caption.
We visualize the result by reshaping the final output back to 38 × 38

Where’s YOUR focus: Personalized Attention

N/A, 2017,

(a) Attention paid more to people and vehicles.

(b) Attention paid more to people than food, even on the back.

(c) Limit attention area to vehicle.

(d) Attention shifts from furniture to electronic devices.

(e) Problematic prediction 1.

(f) Problematic prediction 2.

Figure 8: COCO image, saliency ground truth, model prediction with preference vector:
[outdoor, food, indoor, appliance, sports, person, animal, vehicle, furniture, accessory, electronic, kitchen]
= [0.833, 0.346, 0.189, 0.098, 0.934, 0.679, 0.481, 0.875, 0.081, 0.579, 0.901, 0.223].

(a)

(b)

Figure 9: Effect of fine-tuning. Images are: COCO image, prediction for pvec 1 with original model, prediction for pvec 2 with
original model, prediction for pvec 2 with fine-tuned model.
and then resizing it to the original image shape. We select the
result that can represent the difference between PANet and general
saliency models, which means in the same image there exist objects
belong to multiple super categories, and the user’s preference on
these categories are largely different. From the result we can see that
salient areas of an input image are different in the general saliency
ground truth and in our prediction results, and our prediction area
fit better for this particular input preference.
Performance bottleneck is the accuracy of detection layers. When
detection layers cannot recognize an object or recognizes it wrongly,
predicted saliency map either pays not enough attention to correct
areas, or pays attention to wrong areas. For example, in Figure 8e,
the foreground and background chair belong to the same category,
but model wrongly detects the foreground one as an object with
high preference, thus attracts more attention. Another problem of
the model is that when objects in the preferred category fill the
image, the predicted saliency map will have dispersive attention,
as shown in Figure 8f.

7.3

Same model, different preference

To fit different user-defined mappings without retraining the whole
model each time, we fix the output channel of the Mapping layer to

20, which sets the maximum limit on allowed number of super categories. If the number of user-defined super categories is less than 20,
we automatically fill the remained channels by zero. In our experiment, we test the model with two preference vectors of different
lengths on the SALICON test set (split as in Section 7.1): pvec 1 listed
in Figure 8 and pvec 2 is [person, non−human] = [1.0, 0.05]. First, we
directly pass pvec 2 and new mapping [person → person; others →
non-human] into the original model that is trained with the ground
truth generated upon pvec 1 . The results are shown in Figure 9a3
and 9b3. Comparing with Figure 9a2 and 9b2, “person” ares indeed
get more attention, but still not good enough to reflect pvec 2 , since
pvec 2 is very biased. We then fine-tune the model to see the results.
Loading the weights trained with pvec 1 , and fine-tuning the model
with ground truth generated by pvec 2 as well as changing inputs to
pvec 2 and the new mapping, PANet converges very fast. Only three
epochs (1050 iterations) are needed to make the objective function
reach its lowest point, which is much less than our original training that takes 50K iterations. Figure 9a4 and 9b4 show the results
predicted by the fine-tuned model, which represent pvec 2 much
better. This indicates that given a trained model, doing saliency
predictions on a new preference is easy and fast.
We also measure the performance quantitatively as shown in
Table 2, and the score distributions are shown in Figure 10. When

N/A, 2017,

Sikun LIN and Pan Hui

(a) CC

(b) SIM

(c) KLD

(d) EMD

Figure 10: Performance of original model and fine-tuned model.
Model
Original
Fine-tuned

pvec
pvec 1
pvec 2
pvec 2

CC
0.725
0.689
0.729

SIM
0.742
0.703
0.720

KLD
0.159
0.0120
0.0102

EMD
0.685
0.686
0.591

(a) Average scores.

Center prior
baseline
Detection
baseline

дt 1
дt 2
дt 1
дt 2

CC
0.420
0.431
0.493
0.499

SIM
0.622
0.610
0.587
0.579

KLD
0.316
0.315
11.222
11.086

EMD
3.668
3.635
5.116
5.164

(b) Baseline scores.

Table 2: Performance of original model and fine-tuned
model on different preference vectors.

trying to predict the saliency map for a new preference vector, a
fast fine-tuning will increase the performance significantly than
directly using the original model. The result also demonstrates
that for different preferences, model performances are different
on a same metric, which might be influenced by the variance of
preference scores. KL-divergence score is largely determined by
pvec. An even preference, namely each super category receives
similar attention, will likely result in a dispersive saliency map that
has smaller values at each pixel position, and are more likely to be
penalized by KL-divergence measure.

7.4

Comparison with unbiased models

To see the effect of shifting attention according to personal preference, we compare our model with a state-of-the-art general saliency
prediction model Salicon [15], testing upon the ground truth generated with pvec 1 and pvec 2 as in Section 7.3. We use their codes
[41] to generate fixation maps of the images without any personal
tendency. The results are shown in table 3.
Comparing the predictions of our PANet and generalized models
such as Salicon model, PANet performs much better in fitting user
preference. The saliency maps predicted by a general model such as
Salicon score similarly to the center prior. For different preferences,
the performances will be different but stay in the same level. A
more biased preference will lead to a slightly worse performance
when using the general saliency model.

pvec
CC
SIM KLD EMD
pvec 1 0.527 0.621 0.332 4.605
pvec 2 0.525 0.594 0.337 4.656
Table 3: Salicon model tested on the ground truth generated
with pvec 1 and pvec 2 .

8

FUTURE WORK

Our ground truth generation is based on object bounding box location, which is not accurate enough compared with instance-level
segmentation. However, there is no available segmentation dataset
labeled with object category information. As a future work, a dataset
of instance-level segmented saliency map with categorical information about each object can be collected by labeling existing segmentation datasets. Current model has non-differentiable operation
layers, making the execution time depends on the object numbers
in the input image. Revising the model to a differentiable one will be
the major future work for us. The model can also be extended into
focusing attention on the familiar person when combined with face
recognition models. Another possible direction is extending current
convolutional network into a recurrent version, thus it can find the
personalized pattern about how the viewer changes the attention
areas through a time series, which can be used for personalized
scene description.

9

CONCLUSION

We design, implement, and evaluate a personalized attention prediction model PANet, which can predict saliency areas in an image
according to individual user preference. This work presents the
novel approach of including user preference in the vision model,
allowing more efficient post-processing steps in various HCI applications. The shared feature extraction layers can efficiently provide
multi-scale features for both saliency prediction and preference
fitting stream. For training and validation, we collect saliency labels from two subjects together with their preferences. For new
input preferences, we dynamically generate ground truth from established datasets, with parameters based on collected annotations.
We experimentally validate that our saliency model can fit input
preference and work for different preferences with a fast fine-tuning.
Compared with general saliency prediction models, it fits much
better to the saliency maps biased with personal tendencies. The
more biased a preference is, the more performance gain provided
by PANet.

Where’s YOUR focus: Personalized Attention

REFERENCES
[1] Alexandre Alahi, Raphael Ortiz, and Pierre Vandergheynst. 2012. Freak: Fast
retina keypoint. In Computer vision and pattern recognition (CVPR), 2012 IEEE
conference on. Ieee, 510–517.
[2] Herbert Bay, Tinne Tuytelaars, and Luc Van Gool. 2006. Surf: Speeded up robust
features. In European conference on computer vision. Springer, 404–417.
[3] Christopher JC Burges. 1998. A tutorial on support vector machines for pattern
recognition. Data mining and knowledge discovery 2, 2 (1998), 121–167.
[4] Zoya Bylinskii, Tilke Judd, Ali Borji, Laurent Itti, Frédo Durand, Aude Oliva, and
Antonio Torralba. [n. d.]. MIT Saliency Benchmark. ([n. d.]).
[5] Michael Calonder, Vincent Lepetit, Christoph Strecha, and Pascal Fua. 2010.
Brief: Binary robust independent elementary features. In European conference on
computer vision. Springer, 778–792.
[6] Yanxiang Chen, Gang Tao, Qiangqiang Xie, and Minglong Song. 2016. Video
attention prediction using gaze saliency. Multimedia Tools and Applications (2016),
1–18.
[7] Ming-Ming Cheng, Niloy J Mitra, Xiaolei Huang, Philip HS Torr, and Shi-Min
Hu. 2015. Global contrast based salient region detection. IEEE Transactions on
Pattern Analysis and Machine Intelligence 37, 3 (2015), 569–582.
[8] Gabriella Csurka, Christopher Dance, Lixin Fan, Jutta Willamowski, and Cédric
Bray. 2004. Visual categorization with bags of keypoints. In Workshop on statistical
learning in computer vision, ECCV, Vol. 1. Prague, 1–2.
[9] M. Everingham, L. Van Gool, C. K. I. Williams, J. Winn,
and A. Zisserman. [n. d.].
The PASCAL Visual Object
Classes Challenge 2007 (VOC2007) Results.
http://www.pascalnetwork.org/challenges/VOC/voc2007/workshop/index.html. ([n. d.]).
[10] Pedro Felzenszwalb, David McAllester, and Deva Ramanan. 2008. A discriminatively trained, multiscale, deformable part model. In Computer Vision and Pattern
Recognition, 2008. CVPR 2008. IEEE Conference on. IEEE, 1–8.
[11] Ross Girshick. 2015. Fast r-cnn. In Proceedings of the IEEE International Conference
on Computer Vision. 1440–1448.
[12] Ross Girshick, Jeff Donahue, Trevor Darrell, and Jitendra Malik. 2014. Rich
feature hierarchies for accurate object detection and semantic segmentation. In
Proceedings of the IEEE conference on computer vision and pattern recognition.
580–587.
[13] Stas Goferman, Lihi Zelnik-Manor, and Ayellet Tal. 2012. Context-aware saliency
detection. IEEE Transactions on Pattern Analysis and Machine Intelligence 34, 10
(2012), 1915–1926.
[14] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2014. Spatial pyramid
pooling in deep convolutional networks for visual recognition. In European
Conference on Computer Vision. Springer, 346–361.
[15] Xun Huang, Chengyao Shen, Xavier Boix, and Qi Zhao. 2015. Salicon: Reducing
the semantic gap in saliency prediction by adapting deep neural networks. In
Proceedings of the IEEE International Conference on Computer Vision. 262–270.
[16] Yangqing Jia and Mei Han. 2013. Category-independent object-level saliency
detection. In Proceedings of the IEEE international conference on computer vision.
1761–1768.
[17] Ming Jiang, Shengsheng Huang, Juanyong Duan, and Qi Zhao. 2015. Salicon:
Saliency in context. In Proceedings of the IEEE Conference on Computer Vision and
Pattern Recognition. 1072–1080.
[18] Peng Jiang, Haibin Ling, Jingyi Yu, and Jingliang Peng. 2013. Salient region
detection by ufo: Uniqueness, focusness and objectness. In Proceedings of the
IEEE International Conference on Computer Vision. 1976–1983.
[19] Tilke Judd, Krista Ehinger, Frédo Durand, and Antonio Torralba. 2009. Learning
to predict where humans look. In Computer Vision, 2009 IEEE 12th international
conference on. IEEE, 2106–2113.
[20] Srinivas SS Kruthiventi, Kumar Ayush, and R Venkatesh Babu. 2015. Deepfix: A
fully convolutional neural network for predicting human eye fixations. arXiv
preprint arXiv:1510.02927 (2015).
[21] Matthias Kümmerer, Lucas Theis, and Matthias Bethge. 2014. DeepGaze I: Boosting saliency prediction with feature maps trained on imagenet. arXiv preprint
arXiv:1411.1045 (2014).
[22] Matthias Kümmerer, Thomas SA Wallis, and Matthias Bethge. 2016. DeepGaze
II: Reading fixations from deep features trained on object recognition. arXiv
preprint arXiv:1610.01563 (2016).
[23] Guanbin Li and Yizhou Yu. 2016. Deep contrast learning for salient object
detection. In Proceedings of the IEEE Conference on Computer Vision and Pattern
Recognition. 478–487.
[24] Guanbin Li and Yizhou Yu. 2016. Visual Saliency Detection Based on Multiscale
Deep CNN Features. IEEE Transactions on Image Processing 25, 11 (2016), 5012–
5024.
[25] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva
Ramanan, Piotr Dollár, and C Lawrence Zitnick. 2014. Microsoft coco: Common
objects in context. In European Conference on Computer Vision. Springer, 740–755.
[26] Nian Liu, Junwei Han, Dingwen Zhang, Shifeng Wen, and Tianming Liu. 2015.
Predicting eye fixations using convolutional neural networks. In Proceedings of
the IEEE Conference on Computer Vision and Pattern Recognition. 362–370.

N/A, 2017,
[27] Tie Liu, Zejian Yuan, Jian Sun, Jingdong Wang, Nanning Zheng, Xiaoou Tang, and
Heung-Yeung Shum. 2011. Learning to detect a salient object. IEEE Transactions
on Pattern analysis and machine intelligence 33, 2 (2011), 353–367.
[28] Wei Liu, Dragomir Anguelov, Dumitru Erhan, Christian Szegedy, Scott Reed,
Cheng-Yang Fu, and Alexander C Berg. 2016. SSD: Single shot multibox detector.
In European Conference on Computer Vision. Springer, 21–37.
[29] David G Lowe. 1999. Object recognition from local scale-invariant features. In
Computer vision, 1999. The proceedings of the seventh IEEE international conference
on, Vol. 2. Ieee, 1150–1157.
[30] Federico Perazzi, Philipp Krähenbühl, Yael Pritch, and Alexander Hornung. 2012.
Saliency filters: Contrast based filtering for salient region detection. In Computer
Vision and Pattern Recognition (CVPR), 2012 IEEE Conference on. IEEE, 733–740.
[31] Florent Perronnin, Yan Liu, Jorge Sánchez, and Hervé Poirier. 2010. Large-scale
image retrieval with compressed fisher vectors. In Computer Vision and Pattern
Recognition (CVPR), 2010 IEEE Conference on. IEEE, 3384–3391.
[32] Joseph Redmon, Santosh Divvala, Ross Girshick, and Ali Farhadi. 2016. You
only look once: Unified, real-time object detection. In Proceedings of the IEEE
Conference on Computer Vision and Pattern Recognition. 779–788.
[33] Joseph Redmon and Ali Farhadi. 2016. YOLO9000: Better, Faster, Stronger. arXiv
preprint arXiv:1612.08242 (2016).
[34] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. 2015. Faster r-cnn:
Towards real-time object detection with region proposal networks. In Advances
in neural information processing systems. 91–99.
[35] Edward Rosten, Reid Porter, and Tom Drummond. 2010. Faster and better: A
machine learning approach to corner detection. IEEE transactions on pattern
analysis and machine intelligence 32, 1 (2010), 105–119.
[36] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma,
Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C.
Berg, and Li Fei-Fei. 2015. ImageNet Large Scale Visual Recognition Challenge.
International Journal of Computer Vision (IJCV) 115, 3 (2015), 211–252. https:
//doi.org/10.1007/s11263-015-0816-y
[37] Andrey Rykov. 2016. Port of Single Shot MultiBox Detector to Keras. https:
//github.com/rykov8/ssd_keras.git. (2016).
[38] Xiaohui Shen and Ying Wu. 2012. A unified approach to salient object detection
via low rank matrix recovery. In Computer Vision and Pattern Recognition (CVPR),
2012 IEEE Conference on. IEEE, 853–860.
[39] Karen Simonyan and Andrew Zisserman. 2014. Very deep convolutional networks
for large-scale image recognition. arXiv preprint arXiv:1409.1556 (2014).
[40] Christian Szegedy, Alexander Toshev, and Dumitru Erhan. 2013. Deep neural
networks for object detection. In Advances in Neural Information Processing
Systems. 2553–2561.
[41] Christopher Lee Thomas. 2016. OpenSalicon: An Open Source Implementation of the
Salicon Saliency Model. Technical Report TR-2016-02. University of Pittsburgh.
[42] Jasper RR Uijlings, Koen EA Van De Sande, Theo Gevers, and Arnold WM Smeulders. 2013. Selective search for object recognition. International journal of
computer vision 104, 2 (2013), 154–171.
[43] Paul Viola and Michael Jones. 2001. Robust real-time object detection. International Journal of Computer Vision 4, 34–47 (2001).
[44] Bo Wu and Linfeng Xu. 2014. Integrating bottom-up and top-down visual stimulus
for saliency detection in news video. Multimedia Tools and Applications 73, 3
(2014), 1053–1075. https://doi.org/10.1007/s11042-013-1530-9
[45] Qiong Yan, Li Xu, Jianping Shi, and Jiaya Jia. 2013. Hierarchical saliency detection.
In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.
1155–1162.
[46] Chuan Yang, Lihe Zhang, Huchuan Lu, Xiang Ruan, and Ming-Hsuan Yang. 2013.
Saliency detection via graph-based manifold ranking. In Proceedings of the IEEE
conference on computer vision and pattern recognition. 3166–3173.
[47] Wangjiang Zhu, Shuang Liang, Yichen Wei, and Jian Sun. 2014. Saliency optimization from robust background detection. In Proceedings of the IEEE conference
on computer vision and pattern recognition. 2814–2821.

