1

Predicting Video Saliency with Object-to-Motion
CNN and Two-layer Convolutional LSTM

arXiv:1709.06316v2 [cs.CV] 25 Sep 2017

Lai Jiang, Student Member, IEEE, Mai Xu, Senior Member, IEEE, and Zulin Wang, Member, IEEE

Abstract—Over the past few years, deep neural networks
(DNNs) have exhibited great success in predicting the saliency
of images. However, there are few works that apply DNNs
to predict the saliency of generic videos. In this paper, we
propose a novel DNN-based video saliency prediction method.
Specifically, we establish a large-scale eye-tracking database of
videos (LEDOV), which provides sufficient data to train the
DNN models for predicting video saliency. Through the statistical
analysis of our LEDOV database, we find that human attention
is normally attracted by objects, particularly moving objects
or the moving parts of objects. Accordingly, we propose an
object-to-motion convolutional neural network (OM-CNN) to
learn spatio-temporal features for predicting the intra-frame
saliency via exploring the information of both objectness and
object motion. We further find from our database that there
exists a temporal correlation of human attention with a smooth
saliency transition across video frames. Therefore, we develop
a two-layer convolutional long short-term memory (2C-LSTM)
network in our DNN-based method, using the extracted features
of OM-CNN as the input. Consequently, the inter-frame saliency
maps of videos can be generated, which consider the transition
of attention across video frames. Finally, the experimental results
show that our method advances the state-of-the-art in video
saliency prediction.

I. I NTRODUCTION
A foveated mechanism [1] in the Human Visual System
(HVS) indicates that only small fovea region captures most
visual attention at high resolution, while other peripheral
regions receive little attention at low resolution. To predict
human attention, saliency detection has been widely studied
in recent years, with multiple applications [2] in object recognition, object segmentation, action recognition, image caption,
image/video compression, etc. In this paper, we focus on
predicting video saliency at pixel level, which models attention
on each video frame.
In the early time,the traditional methods of video saliency
prediction mainly follow the integration theory [3]–[7], i.e.,
saliency of video frames can be detected by two steps: (1)
Extract spatial and temporal features from videos for obtaining
conspicuous maps; (2) Conduct a fusion strategy to combine
conspicuous maps of different feature channels together for
generating saliency maps. Benefitting from the state-of-the-art
image saliency prediction, a great deal of spatial features have
L. Jiang, M. Xu and Z. Wang are with the School of Electronic and
Information Engineering, Beihang University, Beijing, 100191 China (e-mail:
Maixu@buaa.edu.cn; jianglai.china@buaa.edu.cn; wzulin@buaa.edu.cn). This
work was supported by the China 973 Program under Grant 2013CB329006,
the National Nature Science Foundation of China projects under Grants
61573037 and 61202139, and the Fok Ying Tung Education Foundation under
Grant 151061. This work was also supported by the MSRA visiting young
faculty program. Mai Xu is the corresponding author of this paper.

been incorporated to predict video saliency [8]–[10]. Additionally, some works focused on designing temporal features
for video saliency prediction, mainly in three aspects: motion
based features [6], [11], [12], temporal contrast features [3],
[4], [13] and compressed domain features [14], [15]. For
fusing spatial and temporal features, many machine learning
algorithms were utilized, such as Support Vector Machine
(SVM) [7], [8], [15], probabilistic model [13], [16], [17] and
phase spectrum analysis [18], [19].
Different from the integration theory, deep natural networks (DNN) based methods have been recently proposed to
learn human attention in an end-to-end manner, significantly
boosting the accuracy of image saliency prediction [20]–
[26]. However, only a few works managed to apply DNN
in video saliency prediction [27]–[29]. Specifically, Cagdas
et al. [27] applied a two-stream CNN structure taking both
RGB frames and motion maps as the inputs, for video saliency
prediction. Bazzani et al. [28] leveraged a deep Convolutional
3D (C3D) network to learn the representations of human
attention on 16 consecutive frames, and then a Long ShortTerm Memory (LSTM) network connected with a mixture
density network was learned to generate saliency maps in
Gaussian mixture distribution. However, the above DNN based
methods1 for video saliency prediction is still in infancy due to
the following drawbacks: (1) Insufficient eye-tracking data for
training DNN; (2) Lack of sophisticated network architecture
simultaneously learning to combine information of object and
motion; (3) Neglect of dynamic pixel-wise transition of video
saliency across video frames.
To avoid the above drawbacks, this paper proposes a new
DNN based method to predict video saliency with spatiotemporal representation and dynamic saliency modeling, benefitting from the analysis of our eye-tracking database. Specifically, we first establish a large-scale video database, which
contains eye-tracking data of 32 subjects on viewing 538
diverse-content videos. Through analysis on our database, we
find that human attention is normally attracted by objects in
a video, especially by the moving objects or moving parts in
the objects. In light of this finding, a novel Object-to-Motion
Convolutional Neural Network (OM-CNN) is constructed to
learn spatio-temporal features for video saliency prediction,
considering both objectness and object motion information.
For dynamic saliency modeling, a Two-layer Convolutional
LSTM (2C-LSTM) network is developed to predict the pixelwise transition of video saliency across frames, with input of
1 None of the above methods makes the code available online. In contrast,
our code is accessible in https://github.com/remega/OMCNN 2CLSTM.

2

spatio-temporal features from OM-CNN. Different from the
conventional LSTM network, this structure is capable of keeping spatial information through the convolutional connections.
To summarize, the main contributions of our work are listed
in the following:
• We establish an eye tracking database consisting of 538
videos in diverse content, with the thorough analysis and
findings on our database.
• We propose the novel OM-CNN structure to predict
saliency of intra-frame, which integrates both objectness
and object motion in a uniform deep structure.
• We develop the 2C-LSTM network with Bayesian
dropout to learn the saliency transition across inter-frame
at pixel-wise level.
The rest of this paper is organized as follows. In Section II,
we briefly review the related works and eye-tracking databases
for video saliency prediction. In Section II-B, we establish
and analyze our large-scale eye-tracking database. According
to the findings on our database, we propose a DNN for video
saliency prediction in Section IV, including both OM-CNN
and 2C-LSTM. Section V shows the experiment results to
validate the performance of our method. Section VI concludes
this paper.
II. R ELATED WORK
In this section, we briefly review the recent works and eyetracking databases for video saliency prediction.
A. Video saliency prediction
Most traditional methods for video saliency prediction [3]–
[7] rely on the integration theory consisting of two main steps:
feature extraction and feature fusion. In the task of image
saliency prediction, many effective spatial features succeed
in predicting human attention with either top-down [8], [30]
or bottom-up [9], [31] strategy. However, video saliency
prediction is more challenging, because temporal features also
play an important role in drawing human attention. To achieve
this, motion based features [6], [11], [12], temporal difference
[3], [4], [13] and compressed domain methods [15], [32] are
widely used in the existing works of video saliency prediction.
Taking motion as additional temporal features, Zhong et al. [6]
proposed to predict video saliency using modified optical flow
with restriction of dynamic consistence. Similarly, Zhou et
al. [12] extended motion feature by computing center motion,
foreground motion, velocity motion and acceleration motion
in their saliency prediction method. In addition to motion,
other methods [3], [4], [13] make use of the temporal changes
in videos for saliency prediction, through figuring out the
contrast between successive frames. For example, Ren et al.
[4] proposed to estimate the temporal difference of each
patch by finding the minimal reconstruction error of sparse
representation over the co-located patches of neighboring
frames. Similarly, in [13], the temporal difference is obtained
via adding pre-designed exponential filters on spatial features
of successive frames. Taking advantage from sophisticated
video coding standards, the compressed domain features are

also explored as spatio-temporal features for video saliency
prediction [15], [32].
In addition to feature extraction, many works focus on the
fusion strategy to generate video saliency maps. Specifically,
a set of probability models were constructed to calculate
the posterior/prior beliefs [16], joint probability distribution
of features [13] and candidate transition probability [17],
in predicting video saliency. Similarly, Li et al. [33] developed a probabilistic multi-task learning method to incorporate
the task-related prior in video saliency prediction. Besides,
other machine learning algorithms, such as SVM and neutral
network, were also applied for linearly [5] or non-linearly
[7] combining the saliency related features. Other advanced
methods [18], [19] apply phase spectrum analysis in the
fusion model to bridge the gap between features and video
saliency. For instance , Guo et al. [18] applied Phase spectrum
of Quaternion Fourier Transform (PQFT) on four feature
channels (two color channels, one intensity channel, and one
motion channel) to predict video saliency.
Most recently, DNN has succeeded in many computer
vision tasks, such as image classification [34], action recognition [35] and object detection [36]. In the field of saliency
prediction, DNN has also been successfully incorporated to
automatically learn spatial features for predicting saliency of
images [21]–[26], [37]. Specifically, as one of the pioneering
works, Deepfix [21] proposed a DNN based structure on
VGG-16 [34] and inception module [38] to learn multi-scales
semantic representation for saliency prediction. In Deppfix,
a dilated convolutional structure was developed to extend
receptive field, and then a location biased convolutional layer
was proposed to learn the centre-bias pattern for saliency
prediction. Similarly, SALICON [22] was also proposed to fine
tune the existing object recognition DNNs, and developed an
efficient loss function for training the DNN model in saliency
prediction. Later, some advanced DNN methods [23]–[25],
[37] were proposed to improve the performance of image
saliency prediction.
However, only a few works manage to apply DNN in video
saliency prediction [27]–[29], [39], [40]. In these DNNs, the
dynamic characteristics were explored in two ways: adding
temporal information in CNN structures [27], [29], [39] or
developing dynamic structure with LSTM [28], [40]. For
adding temporal information, a four-layer CNN in [39] and
a two-stream CNN in [27] were trained, respectively, with
both RGB frames and motion maps as the inputs. Similarly,
in [29], the pair of video frames concatenated with a static
saliency map (generated by the static CNN) are input to the
dynamic CNN for video saliency prediction, allowing CNN to
generalize more temporal features through the representation
learning of DNN. Instead, we find that human attention is more
likely to be attracted by the moving objects or moving parts of
the objects. As such, to explore the semantic temporal features
for video saliency prediction, a motion subnet in our OM-CNN
is trained under the guidance of the objectness subnet.
For developing the dynamic structure, Bazzani et al. [28]
and Liu et al. [40] applied the LSTM networks to predict
human attention, relying on both short- and long-term memory.
However, the fully connections in LSTM limit dimensions of

3

both input and output, unable to obtain end-to-end saliency
map. As such, the strong prior knowledge need to be assumed
for the distribution of saliency. To be more specific, in [28],
the human attention is assumed to distribute as Gaussian
Mixture Model (GMM), then the LSTM is constructed to learn
parameters of GMM. Similarly, [40] focuses on predicting the
saliency of conference videos and assume that the saliency
in each face is a Gaussian distribution. In [40], the face
saliency transition across video frames is learned by LSTM,
and the final saliency map is generated via combining the
saliency of all faces in video. In our work, we first explore
2C-LSTM with Bayesian dropout, to directly predict saliency
maps in an end-to-end manner. This allows learning the more
complex distribution of human attention, rather than preassumed distribution of saliency.
B. Database
The eye-tracking databases of videos collect the fixations of
subjects on each video frame, which can be used as the ground
truth for video saliency prediction. The existing eye-tracking
databases benefit from the mature eye-tracking technology.
In particular, an eye tracker is used to obtain the fixations
of subjects on videos, by tracking the pupil and corneal
reflections [41]. The pupil locations are then mapped to the
real-world stimuli, i.e., video frames, through a pre-defined
calibration matrix. As such, fixations can be located in each
video frame, indicating where people pay attention.
Now, we review the existing video eye-tracking databases.
Table I summarizes the basic properties of these databases. To
the best of our knowledge, CRCNS [42], SFU [43], DIEM [44]
and Hollywood [45] are the most popular databases, widely
used in the most of recent video saliency prediction works [5],
[6], [14], [17], [19], [32], [39], [46], [47]. In the following,
they are reviewed in more details.
CRCNS [42] is one of the earlist video eye-tracking
databases established by Itti et al. in 2004. It is still used as a
benchmark in the recent video saliency prediction works, such
as [46]. CRCNS contains 50 videos mainly including outdoor
scenes, TV shows and video games. The length of each video
ranges from 5.5 to 93.9 seconds, and the frame rate of all
videos is 30 frames per second (fps). For each video, 4 to
6 subjects were asked to look at the main actors or actions.
Afterward, they were required to depict the main content of
video. Thus, CRNS is a task-driven eye-tracking database
for videos. Later, a new database [48] was established, by
manually cutting all 50 videos of CRCNS into 523 “clippets”
with 1-3 second duration, according to the abrupt cinematic
cuts. Another 8 subjects were recruited to view these video
clippets, with their eye-tracking data recorded in [48].
SFU [43] is a public video database containing eye-tracking
data of 12 uncompressed YUV videos, which are frequently
used as the standard test set for video compression and
processing algorithms. Each video is in the CIF resolution
(352 × 288), and is with 3-10 seconds at a frame rate of 30
fps. All eye-tracking data were collected, when 15 non-expert
subjects were free viewing all 12 videos twice.
DIEM [44] is another widely used database, designed to
evaluate the contributions of different visual features on gaze

TABLE I
T HE BASIC PROPERTIES OF THE EXISTING EYE - TRACKING DATABASES .
Database

Year

Videos

Resolution

Duration (s)

Subjects

Mode

CRCNS [42]
SFU [43]
DIEM [44]
Hollywood [45]
Xu [15]
IRCCyN [51]
VAGBA [52]
GazeCom [53]
ASCMN [54]
Coutrot-2 [55]
CAMO [5]
Marat [56]
SAVAM [57]
TUD [58]
Peters [59]
Coutrot-1 [60]

2004
2012
2011
2015
2016
2009
2011
2010
2012
2015
2013
2007
2014
2012
2007
2013

50
12
84
1857
32
51
50
18
24
15
120
53
43
50
24
60

640 × 480
352 × 288
≤ 1280 × 720
≤ 720 × 480
≤ 1920 × 1080
720 × 576
1920 × 1080
1280 × 720
≤ 704 × 576
1232 × 504
640 × 480
720 × 576
1920 × 1080
1280 × 720
640 × 480
720 × 576

6-94
3-10
27-217
2-90
6-25
8-10
10
20
2-76
20-80
1.7-8.6
1-3
18.1 (ave)
20
267
10-24.8

4.7(ave)
15
50(ave)
19
32
37
14
54
13
40
10
15
50
12
5
20

task
free
free
task
free
free
free
free
free
free
free
free
free
task
task
task

clustering. DIEM comprises 84 videos sourced from publicly
accessible videos including advertisement, game trailer, movie
trailer and news clip. Most of these videos have frequent
cinematic cuts. Each video lasts for 27-217 seconds at 30 fps.
The free-viewing fixations of around 50 subjects were tracked
for each video.
Hollywood [45] is a large-scale eye-tracking database for
video saliency prediction, which contains all videos from
two action recognition databases: Hollywood-2 [49] and UCF
sports [50]. All of 1707 videos in Hollywood-2 were selected
from 69 movies, according to 12 action classes, such as
answering phone, eating and shaking hands. UCF sports is
another action database including 150 videos with 9 sport
action classes. The human fixations of 19 subjects were
captured under 3 conditions: free viewing (3 subjects), action
recognition task (12 subjects), and context recognition task (4
subjects). Although the video number of Hollywood is large,
its video content is not diverse, constrained by human actions.
Besides, it mainly focuses on task-driven viewing mode, rather
than free viewing.
As discussed in Section II-B, video saliency prediction
may benefit from the recent development of deep learning.
Unfortunately, as seen in Table I, the existing databases for
video saliency prediction are lack of sufficient eye-tracking
data to train DNN. Although Hollywood [45] has 1857 videos,
it mainly focuses on task-driven visual saliency. Besides, the
video content of Hollywood is limited, only involving human
actions of movies. In fact, a large-scale eye-tracking database
for video should have 3 criteria: 1) a large number of videos, 2)
sufficient subjects, and 3) various video content. In this paper,
we establish a large-scale eye-tracking database of videos,
satisfying the above three criteria. The detail of our large-scale
databases is to be discussed in Section III.
III. D TABASE
In this section, a new Large-scale Eye-tracking Database
of Videos (LEDOV) is established, which is available online
for facilitating the future research. More details and analysis
about our LEDOV database are discussed in the following.
A. Databases establishment
We present our LEDOV database from the aspects of
stimuli, apparatus, participant and procedure.

4

TABLE II
C ATEGORIES OF VIDEOS IN OUR DATABASE BASED ON CONTENT.

Number of sub-classes*
Number of videos
*

Daily action
23
74

Sports
17
58

Human
Social activity
21
69

Art performance
19
59

Animal

Man-made object

Overall

51
156

27
122

148
538

Here, we also report the number of sub-class in each class of videos. For example, in animal videos, the sub-classes include penguin, rabbit, elephant,
etc.

Fig. 1. Examples for each class of videos in LEDOV. From left to right, the videos belong to daily action, sports, social activity, art performance, animal
and man-made object.

TABLE III
N UMBERS OF VIDEOS WITH DIFFERENT AMOUNT OF OBJECTS .
Object number
Video amount

=1
377

=2
103

>2
58

Overall
538

Stimuli. 538 videos, in total 179,336 frames and 6,431 seconds, were collected, according to the following four criteria.
1) Diverse video content. Videos with diverse content were
sourced from different online accessible repositories,
such as daily vlogs, documentaries, movies, sport casts,
TV shows, etc. In Table II, we briefly classify all 538
videos according to their content. Some examples are
shown in Figure 1.
2) Including at least one object. Only videos with at least
one object were qualified for our database. Table III
reports the numbers of videos with different amount of
objects in our database.
3) High quality. We ensured high quality of videos in our
database by choosing those with at least 720p resolution
and 24 Hz frame rate. To avoid quality degradation, the
bit rates were maintained when converting videos to the
uniform MP4 format.
4) Stable shot. The videos with unsteady camera motions and frequent cinematic cuts were not included in
LEDOV. Specifically, there are 212 videos with stable
camera motion. Other 316 videos are without any camera motion.
Apparatus&Participants. For monitoring the binocular eye
movements, an eye tracker, Tobii TX300 [61], was used in
our experiment. TX300 is an integrated eye tracker with a 23”
TFT monitor at screen resolution of 1920 × 1080. During the
experiment, TX300 captured gaze data at 300 Hz. According
to [61], the gaze accuracy can reach 0.4 vision degree (around
15 pixels in stimuli) in the ideal working condition2 . For
more details about Tobii TX300, refer to [61]. Moreover,
32 participants (18 males and 14 females), aging from 20
2 The ideal condition is that the illumination in working environment is
constant at 300 lux, and that the distance between subjects and the monitor is
fixed at 65 cm. Such condition was satisfied in our eye-tracking experiment.

to 56 (32 in average), were recruited to take part in the
eye-tracking experiment. All participants were non-expert for
the eye-tracking experiment, with normal/corrected-to-normal
vision. It is worth pointing out that only those who passed
the calibration of the eye tracker and had less than 10%
fixation dropping rate, were quantified for our eye tracking
experiment. As a result, 32 among 60 subjects were selected
in our experiment.
Procedure. Since visual fatigue may arise after viewing
videos for a long time, 538 videos in LEDOV were equally
divided into 6 non-overlapping groups with similar numbers
of videos in content (i.e., human, animal and man-made
object). During the experiment, each subject was seated on
an adjustable chair around 65 cm from the screen, followed
by a 9-point calibration. Then, the subject was required to
free view 6 groups of videos in a random order. In each
group, the videos were also displayed at random. Between
two successive videos, we inserted a 3-second rest period with
black screen and a 2-second guidance image with a red circle
in screen center. As such, the eyes can be relaxed, and then
the initial gaze location can be reset at center. After viewing
a group of videos, the subject was asked to take a break until
he/she was ready for viewing the next group of videos. Finally,
5,058,178 fixations (saccades and other eye movements have
been removed) were recorded from 32 subjects on 538 videos,
for our LEDOV database.
B. Database analysis
In this section, we mine our database to analyze human
attention on videos. More details are introduced as follows.
1) Temporal correlation of attention on consecutive frames:
It is interesting to explore the temporal correlation of attention
across consecutive frames. In Figure 2, we show human
fixation maps along with some consecutive frames, for 3
selected videos. As we can see from Figure 2, there exists high
temporal correlation of attention across consecutive frames of
videos. To quantify such correlation, we further measure the
linear correlation coefficient (CC) of fixation map between two
consecutive frames. Assume that Gc and Gp are fixation maps
of current and previous frames. Then, the CC value of fixation
maps averaged over a video can be calculated as follows,

5

(a)

2s before

(b)

1.5s before

(c)

1s before

(d)

0.5s before

(e)

Current frame

Fig. 2. Examples for human fixation maps across consecutive frames.

1

Correlation coefficient

0.9
0.8
0.7
0.6
0.5

0-0.5s before

(a)

Current frame

(b)

One candidate

(c)

Multiple candidates

(d)

Optical flow

Fig. 4. Examples of ground-truth fixation maps and candidate objects detected
by YOLO. (a) shows randomly selected frames from three videos in our
LEDOV database. (b) illustrates the fixation maps as well as one candidate
object in each frame. (c) demonstrates fixation maps and multiple candidate
objects. (d) displays optical flow maps of each frame, represented in HSV
color space.

0.5-1s before
1-1.5s before

0.4

numbers of candidate objects in YOLO, via setting thresholds
of confidence probability and non-maximum suppression. Figure 4-(b) shows the examples of one detected object, while
0.1
0
Figure 4-(c) shows the results for more than one object.
Animal
Human
Man-made
All
We can observe from Figure 4-(b) that attention is normally
attended to object regions. We can also see from Figure 4(c) that more human fixations can be included along with
Fig. 3. The CC results of temporal attention correlation, averaged over animal,
increased number of detected candidate objects. To quantify
human, man-made object and all videos in LEDOV.
the correlation between human attention and objectness, we
measure the proportion of fixations falling into object regions
to those of all regions. In Figure 5-(a), we show the fixation
proportion at increased number of candidate objects, averaged
1 X 1 X
Cov(N (Gc ), N (Gp ))
CC =
, over all videos in LEDOV. We can observe from this figure that
|Vc |
|Vp |
Std(N (Gc )) · Std(N (Gp ))
c∈Vc
p∈Vp
fixation proportion hitting on object regions is much higher
Gc − Mean(Gc )
where
N (Gc ) =
.
(1) than that hitting on random region. This implies that there
Std(Gc )
exists high correlation between objectness and human attention
when viewing videos. Figure 5-(a) also shows that fixation
In (1), Vc is the set of all frames in the video, while Vp
proportion increases alongside more candidate objects, which
is the set of consecutive frames before frame c. Additionally,
indicates that human attention may be attracted by more than
Cov(·), Std(·) and Mean(·) are covariance, standard deviation
one object.
and mean operators. For Vp , we choose 4 sets of previous
In addition, one may find from Figure 4 that human attention
frames, i.e., 0-0.5s before, 0.5-1s before, 1-1.5s before and
is attended to only small parts of object regions. Therefore, we
1.5-2s before. Then, in Figure 3, we plot the CC results of
measure the proportion of fixation area3 inside the object to
these 4 sets of Vp , which are averaged over all videos in our
the whole object area. Figure 5-(b) shows the results of such
LEDOV database. We also show in Figure 3 one-vs-all results,
proportion, at increased number of detected candidate objects.
which is the baseline of averaged CC between fixation maps
We can see from this figure that the proportion of fixation area
of one subject and the rest (indicating attention correlation
proportion decreases as at more candidate objects.
between humans). We can see from this figure that the CC
3) Correlation between motion and human attention: From
value of temporal consistency is much higher than that of oneour
LEDOV database, we find that human attention trends
vs-all baseline. This implies high temporal correlation across
to
focus
on moving objects or the moving parts of objects.
consecutive frames of video. We can further find that temporal
Specifically,
as shown in the first row of Figure 6, human
correlation of attention decreases, when increasing the distance
attention
is
transited
to the big penguin, when it suddenly
between the current and previous frames. Consequently, the
falls
with
a
rapid
motion.
Besides, the second row of Figure
long- and short-term dependency of attention across frames of
6
shows
that,
in
the
scene
with a single salient object, the
video can be verified.
intensive moving parts of the player may considerably attract
2) Correlation between objectness and human attention: It
more fixations than other parts. It is interesting to further
is intuitive that people may be attracted by objects rather than
explore the correlation between motion and human attention
background when watching videos. Therefore, we investigate
how much attention is related to object regions. First, we apply
3 The fixation area is computed by a pre-set threshold on the fixation map.
a CNN-based objection detection method YOLO [36] to detect The fixation map is generated by the fixation points convolved with a Gaussian
main objects in each video frame. Here, we generate different filter as [62].
0.3
0.2

1.5-2s before
One-vs-all

6

Fixation area in object regions (proportion)

0.4

Fixation proportion to all fixations

0.7
0.6
0.5
All
Animal
Human
Man-made
Random region

0.4
0.3
0.2
0.1
0

1

2

3

4

5

6

7

8

9

10

All
Animal
Human
Man-made
Random region

0.35
0.3
0.25
0.2
0.15
0.1
0.05
0

1

2

3

Candidate object number

4

5

6

7

8

9

10

Candidate object number

(a) Fixation hit proportion

(b) Fixation area proportion

Fig. 5. (a) Fixation proportion belonging to object regions at increased numbers of detected candidate objects. (b) Proportion of fixation area in object regions
along with increased numbers of detected candidate objects. The results of all videos as well as animals, human and man-made object videos are plotted with
different curves. Besides, the results of fixations hitting on random region are plotted as the baseline.

Fig. 6. Attention heat maps of some frames selected from video animal penguin07 and human basketball02, where the human attention is attracted by
moving objects or the moving parts of objects.

IV. P ROPOSED METHOD

100%
80%
60%

6.6%
9.1%
15.3%

7.6%
11.0%

7.1%
9.2%
13.8%

7.3%
10.2%
15.9%

44.9%

All

16.8%
40%
20%

50.2%

41.4%

49.6%

Animal

Human

Man-made

0%

The rank of motion intensity
0-10%

10%-20%

20%-30%

30%-40%

40%-50%

50%-60%

60%-70%

70%-80%

80%-90%

90%-100%

Fig. 7. Proportion of fixations belonging to 10 groups, ranked according to
the motion intensity.

inside the regions of objects. Here, we apply FlowNet [63],
a CNN based optical flow method, to measure the motion
intensity in all frames (some results are shown in Figure
4-(d)). At each frame, pixels are ranked according to the
descending order of motion intensity. Subsequently, we cluster
the ranked pixels into 10 groups with equal number of pixels,
over all video frames in the LEDOV database. For example,
the first group includes pixels with top 10% ranked motion
intensity. The numbers of fixations falling into each group
are shown in Figure 7. We can see from Figure 7 that 44.9%
fixations belong to the group with top 10% high-valued motion
intensity. This implies the high correlation between motion and
human attention within the region of objects.

A. Framework
For video saliency prediction, we develop a new DNN
architecture that combines OM-CNN and 2C-LSTM together.
Inspired by the second and third findings of Sections III-B,
OM-CNN integrates both regions and motions of objects to
predict video saliency through two subnets, i.e., the subnets
of objectness and motion. In OM-CNN, the objectness subnet
yields a coarse objectness map, which is used to mask the
features output from the convolutional layers in the motion
subnet. Then, the spatial features from the objectness subnet
and temporal features from the motion subnet are concatenated
to generate spatio-temporal features of OM-CNN. The architecture of OM-CNN is shown in Figure 8. According to the
first finding of Section III-B, 2C-LSTM with Bayesian dropout
is developed to learn dynamic saliency of video clips, in which
the spatio-temporal features of OM-CNN work as the input.
Finally, the saliency map of each frame is generated from 2
deconvolutional layers of 2C-LSTM. The architecture of 2CLSTM is shown in Figure 9.
In the following, we present the detail of OM-CNN via
introducing the objectness subnet in Section IV-B and the
motion subnet in Section IV-C. In addition, the detail of 2CLSTM is discussed in Section IV-D. The training process of
our DNN method is presented in Section IV-E.
B. Objectness subnet in OM-CNN
The second finding of Section III-B has shown that objects
draw extensive attention in videos. Therefore, OM-CNN in-

7

(a) The overall architecture of OM-CNN

(b) The details for sub-modules of inference module and feature normalization
Fig. 8. Overall architecture of our OM-CNN for predicting video saliency of intra-frame. The sizes of convolutional kernels are shown in the figure. For
instance, 3 × 3 × 16 means 16 convolutional kernels with size of 3 × 3. Note that the 7 − 9th convolutional layers (C1 7 ,C1 8 &C1 9 ) in the objectness
subnet have the same size of convolutional kernels, thus sharing the same cube in (a) but not sharing the parameters. Similarly, each of the last four cubes in
the motion subnet represents 2 convolutional layers with same kernel size. The details of the inference and feature normalization modules are shown in (b).

cludes an objectness subnet, for extracting multi-scale spatial
features related to objectness information. The basic structure
of the objectness subnet is based on a pre-trained YOLO
[36]. Note that YOLO is a state-of-the-art CNN architecture,
capable of detecting video object with high accuracy. In OMCNN, the YOLO structure is utilized to learn spatial features
of the input frame for saliency prediction. To avoid overfitting, the fast version of YOLO is applied in the objectness
subnet, including 9 convolutional layers, 5 pooling layers
and 2 Fully Connected layers (FC). To further avoid overfitting, an additional batch-normalization layer is added to
each convolutional layer. Assuming that P (·) and ∗ are the
max pooling and convolution operations, the output of the kth convolutional layer Cko in the objectness subnet can be
computed as
Cko = L0.1 (P (Ck−1
) ∗ Wok−1 + Bk−1
),
o
o

(2)

where Wok−1 and Bk−1
indicates the kernel parameters of
o
weight and bias at the (k − 1)-th convolutional layer. Additionally, L0.1 (·) is a leaky ReLU activation with leakage
coefficient of 0.1.
For leveraging the multi-scale information with various
receptive fields, feature normalization (FN) operation is in-

troduced in OM-CNN to normalize and concatenate certain
convolutional layers of the objectness subnet. As shown in
Figure 8-(b), FN includes a 1 × 1 convolutional layer and a
bilinear layer to normalize the input features into 128 channels
with size of 28 × 28. Specifically, in the objectness subnet,
the outputs of the 4-th, 5-th, 6-th and last convolutional layer
are normalized by FN to obtain 4 sets of spatial features
{FSi }4i=1 , referring to multiple scales. Besides, the output
of the last FC layer in YOLO indicates the sizes, class
probabilities and confidence of candidate objects in each grid.
Then, the output of the last FC layer needs to be reshaped
into a spatial feature with size of 7 × 7 × 30 (representing 30
channels with the size of 7 × 7). After bilinearly interpolating
on the 7 × 7 × 30 spatial feature, the high level spatial feature
FS5 can be obtained with size of 28×28×30. At last, the final
spatial features are generated by concatenating {FSi }5i=1 .
Given the spatial features {FSi }5i=1 , an inference module
Ic is designed to generate a coarse objectness map Sc :
Sc = Ic ({FSi }5i=1 ).

(3)

The inference module Ic is a CNN structure consisting of
4 convolutional layers and 2 deconvolutional layers with
stride of 2. The architecture of Ic is shown in Figure 8-(b).

8

Consequently, the coarse objectness map Sc can be obtained
to encode the objectness information, roughly related to salient
regions.
C. Motion subnet in OM-CNN
Next, a motion subnet, also shown in Figure 8-(a), is
incorporated in OM-CNN to extract multi-scale temporal
features from the pair of neighboring frames. According to
the third findings of Section III-B, attention is more likely
to be attracted by the moving objects or moving parts of the
objects. Therefore, following the object subnet, the motion
subnet is developed to extract motion features within object
regions. The motion subnet is based on FlowNet [63], a CNN
structure to estimate optical flow. In the motion subnet, only
the first 10 convolutional layers of FlowNet are applied, in
order to reduce the number of parameters. To model motion
in object regions, the coarse objectness map Sc in (3) is used to
mask the outputs of first 6 convolutional layers of the motion
subnet. As such, the output of the k-th convolutional layer
Ckm can be computed as
k−1
k−1
Ckm = L0.1 (M (Ck−1
m , Sc ) ∗ Wm + Bm ),

where

k−1
M (Ck−1
m , Sc ) = Cm · (Sc · (1 − γ) + 1 · γ). (4)

k−1
indicate the kernel parameters of
and Bk−1
In (4), Wm
m
weight and bias at the (k − 1)-th convolutional layer in the
motion subnet; γ (0 ≤ γ ≤ 1) is the adjustable parameter to
control the mask degree, mapping the range of Sc from [0, 1] to
[γ, 1]. Note that the last 4 convolutional layers are not masked
with the coarse objectness map, for considering the motion of
non-object region in saliency prediction. Afterwards, similar
to the objectness subnet, the outputs of the 4-th, 6-th, 8-th and
10-th convolutional layers are computed by FN, such that 4
sets of temporal features {FTi }4i=1 with size of 28 × 28 × 128
are achieved.
Then, given the extracted features {FSi }5i=1 and {FTi }4i=1
from two subnets of OM-CNN, another inference module If
is constructed to generate a fine saliency map Sf , modeling
the intra-frame saliency of a video frame. Mathematically, Sf
can be computed as

Sf =

If ({FSi }5i=1 , {FTi }4i=1 ).

Fig. 9. Architecture of our 2C-LSTM for predicting saliency transition across
inter-frame, following the OM-CNN. Note that the training process is not
annotated in the figure.

t−1
input features, through the memory cells (Mt−1
1 , M2 ) and
t−1
t−1
hidden states (H1 , H2 ) of 1-st and 2-nd LSTM layers
at the last frame. Then, the hidden states of the 2-nd LSTM
layer Ht2 are fed into 2 deconvolutional layers to generate final
saliency map Stl at frame t. The architecture of 2C-LSTM is
shown in Figure 9.

The same as [64], we extend the conventional LSTM via replacing Hadamard product (denoted as ◦) by the convolutional
operator (denoted as ∗), in order to consider spatial correlation
of input OM-CNN features in the dynamic model. Taking the
first LSTM layer as example, a single LSTM cell at frame t
can be written as
It1

= σ(Ht−1
∗ Wih + Ftst ∗ Wif + Bi ),
1

At1

= σ(Ht−1
∗ Wah + Ftst ∗ Waf + Ba ),
1

Ot1

= σ(Ht−1
∗ Woh + Ftst ∗ Wof + Bo ),
1

Gt1

=

Mt1

= Ft1 ◦ M1t−1 + It1 ◦ Gt1 ,

Ht1

= Ot1 ◦ tanh(Mt1 ),

tanh(Ht−1
∗ Wgh + Ftst ∗ Wgf + Bg ),
1
(6)

(5)

Here, Sf is also used to train the OM-CNN model, to be
discussed in Section IV-E. Besides, the architecture of If is
same as that of Ic . As shown in Figure 8-(b), in If , the output
of convolutional layer C4 with size of 28×28×128 is viewed
as the final spatio-temporal features, denoted as Fst , to predict
intra-frame saliency. Next, Fst is fed into 2C-LSTM, to be
presented in the following.
D. Convolutional LSTM

where σ and tanh are the activation functions
of
sigmoid
and
hyperbolic
tangent.
In
(6),
{Wih , Wah , Woh , Wgh , Wif , Waf , Wof , Wgf }
and
{Bi , Ba , Bo , Bg } denote the kernel parameters of weight
and bias at the corresponding convolutional layer; It1 , At1 and
Ot1 are the the gates of input, forget and output for frame t;
Gt1 , Mt1 and Ht1 are the input modulation, memory cells and
hidden states. They are all represented by 3-D tensors with
size of 28 × 28 × 128 in 2C-LSTM.

In this section, we develop the 2C-LSTM network for learning to predict dynamic saliency of a video clip, since the first
finding of Section III-B illustrates that there exists dynamic
transition of attention across video frames. At frame t, taking
the OM-CNN features Fst as the input (denoted as Ftst ), 2CLSTM leverages both long- and short-term correlation of the

In our method, we adopt two-layer LSTM cells to learn temporal correlation of high-dimensional features (28 × 28 × 128).
On the other hand, the two-layer LSTM cells decrease the
generalization ability. Thus, to improve generalization ability,
we apply the Bayesian inference based dropout [65] in each
convolutional LSTM cell. Then, the LSTM cell of (6) can be

9

TABLE IV
T HE VALUES OF HYPER - PARAMETERS IN OM-CNN AND 2C-LSTM.

rewritten as
It1
At1
Ot1
Gt1
Mt1
Ht1

= σ((Ht−1
◦ Zhi ) ∗ Wih + (Ftst ◦ Zfi ) ∗ Wif + Bi ),
1
=
=
=
=
=

σ((Ht−1
1
σ((Ht−1
1

◦ Zha ) ∗ Wah + (Ftst ◦ Zfa ) ∗ Waf + Ba ),
◦ Zho ) ∗ Woh + (Ftst ◦ Zfo ) ∗ Wof + Bo ),
tanh((Ht−1
◦ Zhg ) ∗ Wgh + (Ftst ◦ Zfg ) ∗ Wgf + Bg ),
1
Ft1 ◦ Mt−1
+ It1 ◦ Gt1 ,
1
Ot1 ◦ tanh(Mt1 ).
(7)

In (7), {Zhi , Zha , Zho , Zhh } and {Zfi , Zfa , Zfo , Zfg } are 2 sets of
random masks for the hidden states and input features before
convolutional operation. These masks are generated by a Ltimes Monte Carlo integration, with hidden dropout rate ph
and feature dropout rate pf , respectively.
E. Training process
For training OM-CNN, we utilize the Kullback-Leibler
(KL) divergence based loss function to update the parameters.
It is because [22] has proved that the KL divergence is
more effective than other metrics in training DNN to predict
saliency. Regarding the saliency map as a probability distribution of attention, we can measure the KL divergence DKL
between fine saliency map Sf of OM-CNN and ground truth
distribution G of human fixations as follows,
DKL (G, Sf ) = G log

G
.
Sf

(8)

In (8), the smaller KL divergence indicates higher accuracy in
saliency prediction. Furthermore, the KL divergence between
coarse objectness map Sc of OM-CNN and ground truth G is
also used as an auxiliary function to train OM-CNN. This is
based on the assumption that the object regions are correlated
with salient regions. Then, the OM-CNN model is trained by
minimizing the following loss function,
LOM−CNN =

1
λ
DKL (G, Sf ) +
DKL (G, Sc ). (9)
1+λ
1+λ

In (9), λ is a hyper-parameter to control the weights of two KL
divergences. Note that OM-CNN is pre-trained on YOLO and
FlowNet, and the rest parameters of OM-CNN are initialized
by the Xavier initializer [66].
To train 2C-LSTM, the training videos are cut into clips
with the same length T . Besides, when training 2C-LSTM,
the parameters of OM-CNN are fixed to extract the spatiotemporal features of each T -frame video clip. Then, the
loss function of 2C-LSTM is defined as the averaged KL
divergence over T frames
L2C−LSTM

T
1X
DKL (Sil , Gi ).
=
T i=1

(10)

In (10), {Sil }Ti=1 are the final saliency maps generated by 2CLSTM, and {Gi }Ti=1 are the ground truth of attention maps.
For each LSTM cell, the kernel parameters are initialized by
the Xavier initializer, while the memory cells and hidden states
are initialized by zeros.

Hyper-parameters in OM-CNN

Hyper-parameters in 2C-LSTM

Objectness mask parameter γ in (4)
KL divergences weight λ in (9)
Initial learning rate
Training epochs (iterations)
Batch size
Weight decay
Bayesian dropout rates ph and pf in 2C-LSTM
Times of Monte Carlo integration L in in 2C-LSTM
Initial learning rate
Training epochs (iterations)
Weight decay

0.5
0.5
1 × 10−5
12(∼ 1.5 × 105 )
12
5 × 10−6
0.25&0.25
100
1 × 10−4
15(∼ 2 × 105 )
5 × 10−6

V. E XPERIMENT
In this section, the experimental results are presented to
validate the performance of our method in video saliency
prediction. Section V-A introduces the settings in our experiments. Section V-B and V-C compare the accuracy of
saliency prediction on LEDOV and other 2 public databases,
respectively. Furthermore, the results of ablation experiment
are discussed in Section V-D, to analyze the effectiveness of
each individual component proposed in our method.
A. Settings
In our experiment, 538 videos in the LEDOV database are
randomly divided into training (436 videos), validation (41
videos) and test (41 videos) sets. Specifically, to learn 2CLSTM, we temporally segment 456 training videos into 24,685
clips, all of which have T = 16 frames. The overlap of 10
frames is allowed in cutting video clips, for the purpose of
data augmentation. Before inputting to OM-CNN, the RGB
channels of each frame are resized to 448 × 448, with their
mean values being removed. In training OM-CNN and 2CLSTM, we learn the parameters using the stochastic gradient
descent algorithm with Adam optimizer [67]. Here, the hyperparameters of OM-CNN and 2C-LSTM are tuned to minimize
the KL divergence of saliency prediction over the validation
set. The tuned values of some key hyper-parameters are listed
in Table IV. Given the trained models of OM-CNN and 2CLSTM, all 41 test videos in LEDOV are used to evaluate the
performance of our method, compared with other 8 state-ofthe-art methods. All experiments are conducted on a computer
with Intel(R) Core(TM) i7-4770 CPU@3.4 GHz, 16 GB RAM
and a single GPU of Nvidia GeForce GTX 1080. Benefitting
from the acceleration of GPU, our method is able to make the
real-time prediction for video saliency at a speed of 30 fps.
B. Evaluation on LEDOV
In this section, we compare the accuracy of video saliency
predicted by our and other 8 state-of-the-art methods, including GBVS [11], PQFT [18], Rudoy [17], OBDL [14],
SALICON [22], Xu [15], BMS [10] and SalGAN [26]. Among
them, [11], [18], [17], [14] and [15] are 5 state-of-the-art
saliency prediction methods for videos. Besides, we compare
two DNN based methods, [22] and [26]. In our experiments,
we apply four metrics to measure the accuracy of saliency
prediction: the area under receiver operating characteristic
curve (AUC), normalized scanpath saliency (NSS), CC, and
KL divergence. Note that the larger value of AUC, NSS or

10

TABLE V
M EAN ( STANDARD DEVIATION ) OF SALIENCY PREDICTION ACCURACY FOR OUR AND OTHER 8 METHODS OVER ALL TEST VIDEOS IN LEDOV.

AUC
NSS
CC
KL

Our

GBVS [11]

PQFT [18]

Rudoy [17]

OBDL [14]

SALICON [22]

Xu [15]

BMS [10]

SalGAN [26]

0.90(0.04)
2.94(0.85)
0.57(0.12)
1.24(0.39)

0.84(0.06)
1.54(0.74)
0.32(0.13)
1.82(0.39)

0.70(0.08)
0.69(0.46)
0.14(0.08)
2.46(0.39)

0.80(0.08)
1.45(0.64)
0.32(0.14)
2.42(1.53)

0.80(0.09)
1.54(0.84)
0.32(0.16)
2.05(0.74)

0.85(0.06)
2.33(0.87)
0.44(0.13)
1.64(0.42)

0.83(0.06)
1.47(0.47)
0.38(0.11)
1.65(0.30)

0.76(0.09)
0.98(0.48)
0.21(0.09)
2.23(0.39)

0.87(0.06)
2.19(0.59)
0.43(0.09)
1.68(0.33)

Fig. 10. Saliency maps of 8 videos randomly selected from the test set of LEDOV. The maps were yielded by our and other 8 methods as well the ground-truth
human fixations. Note that the results of only one frame are shown for each selected video.

CC indicates more accurate prediction of saliency, and the
smaller KL divergence means better saliency prediction. Table
V tabulates the results of AUC, NSS, CC and KL divergence
for our and other 8 methods, which are averaged over 41 test
videos of the LEDOV database. We can see from this table that
our method performs much better than all other methods in
all 4 metrics. More specifically, our method achieves at least
0.03, 0.61, 0.13 and 0.40 improvements in AUC, NSS, CC
and KL. Besides, two DNN based methods, SALICON [22]
and SalGAN [26], outperform other conventional methods.
This verifies the effectiveness of saliency related features
that are automatically learned by DNN, better than hand
crafted features. On the other hand, our method is significantly
superior to [22] and [26]. The main reasons are as follows:
(1) Our method embeds the objectness subnet to make use of
objectness information in saliency prediction; (2) The object
motion is explored in the motion subnet to predict video
saliency. (3)The network of 2C-LSTM is leveraged to model
saliency transition across video frames. Section V-D analyzes

the above three reasons with more details.

Next, we move to the comparison of subjective results in
video saliency prediction. Figure 10 demonstrates the saliency
maps of 8 randomly selected videos in test set, detected by our
and other 8 methods. In this figure, one frame is selected for
each video. One may observe from Figure 10 that our method
is capable of well locating the salient regions, much closer to
the ground-truth maps of human fixations. In contrast, most of
other methods fail to accurately predict the regions attracting
human attention. In addition, Figure 11 shows the saliency
maps of some frames selected from one test video. As seen
in this figure, our method is able to model human fixation
with smooth transition than other 8 methods. In summary,
our method is superior to other state-of-the-art methods in
both objective and subjective results, tested on our LEDOV
database.

11

Fig. 12. Saliency prediction accuracy of objectness subnet, motion subnet,
OM-CNN and the combination of OM-CNN and 2C-LSTM, compared with
SALICON [22] and SalGAN [26]. Note that the smaller KL divergence
indicates higher accuracy in saliency prediction.

Fig. 11. Saliency maps of several frames randomly selected from a single
test video in LEDOV. The maps were yielded by our and other 8 methods as
well the ground-truth human fixations.

C. Evaluation on other databases
To evaluate the generalization capability of our method, we
further compare the performance of our and other 8 methods
on two widely used databases SFU [43] and DIEM [44], which
are available online. The same as [14], 20 videos of DIEM and
all videos of SFU are tested to assess the saliency prediction
performance. In our experiments, the models of OMM-CNN
and 2C-LSTM, learned from the training set of LEDOV, are
directly used to predict saliency of test videos from the DIEM
and SFU databases. Table VI presents the averaged results of
AUC, NSS, CC and KL for our and other 8 methods over
SFU and DIEM, respectively. We can see from this table that
our method again outperforms other 8 methods, especially
in the DIEM database. In particular, there are at least 0.04,
0.44, 0.09 and 0.25 improvements in AUC, NSS, CC and KL.
Such improvements are comparable to those in our LEDOV
database. This implies the generalization capability of our
method in video saliency prediction.
D. Performance analysis of saliency prediction
Since the OM-CNN architecture of our method is composed
of the objectness and motion subnets, we evaluate the contribution of each single subnet. We further analyze the contribution
of 2C-LSTM, by comparing the trained models with and
without 2C-LSTM. Specifically, the objectness subnet, motion
subnet and OM-CNN are trained independently, with the
same settings introduced above. Then, they are compared
with our method, i.e., the combination of OM-CNN and 2C-

LSTM. The comparison results are shown in Figure 12. We
can see from this figure that OM-CNN performs better than
the objectness subnet with 0.05 reduction in KL divergence,
and outperforms the motion subnet with 0.09 KL divergence
reduction. This indicates the effectiveness of integrating the
subnets of objectness and motion. Besides, the combination
of OM-CNN and 2C-LSTM reduces the KL divergence by
0.09, over the single OM-CNN architecture. Hence, we can
conclude that 2C-LSTM can further improve the performance
of OM-CNN, due to exploring temporal correlation of saliency
across video frames.
Furthermore, we analyze the performance of Bayesian
dropout in 2C-LSTM, which aims to avoid the over-fitting
caused by the high dimensionality of 2C-LSTM. Through the
experimental results, we find that the low dropout rate may
incur the under-fitting issue, resulting in accuracy reduction
of saliency prediction. To analyze the impact of the dropout
rate, we train the 2C-LSTM models at different values of
hidden dropout rate ph and feature dropout rate pf . The trained
models are tested over the validation set of LEDOV, and the
averaged results of KL divergence are shown in Figure 13.
We can observe from the figure that the Bayesian dropout can
bring around 0.03 KL reduction when both ph and pf are set
to 0.25. However, the KL divergence sharply rises, once ph
and pf are increased from 0.25 to 1. Therefore, we set ph and
pf to be 0.25 in our method. They may be adjusted according
to the amount of training data.
VI. C ONCLUSION
In this paper, we have proposed a DNN based video saliency
prediction method. In our method, two DNN architectures
were developed, i.e., OM-CNN and 2C-LSTM. The two DNN
architectures were driven by the LEDOV database established
in this paper, which is composed of 32 subjects fixations on
538 videos. Interestingly, we found from the LEDOV database

12

TABLE VI
M EAN ( STANDARD DEVIATION ) VALUES FOR SALIENCY PREDICTION ACCURACY OF OUR AND OTHER METHODS OVER SFU AND DIEM DATABASES .
SFU
Our
GBVS [11]
PQFT [18]
Rudoy [17]
OBDL [14]
SALICON [22]
Xu [15]
BMS [10]
SalGAN [26]
AUC 0.81(0.07)
0.76(0.07)
0.61(0.09)
0.73(0.08)
0.74(0.10)
0.78(0.08)
0.80(0.07)
0.66(0.08)
0.79(0.07)
NSS
1.46(0.65)
0.91(0.47)
0.31(0.34)
0.83(0.45)
1.03(0.64)
1.24(0.60)
1.24(0.39)
0.50(0.31)
1.25(0.47)
CC
0.55(0.15)
0.44(0.15)
0.12(0.15)
0.34(0.15)
0.42(0.21)
0.58(0.22)
0.43(0.12)
0.25(0.11)
0.51(0.13)
KL
0.67(0.24)
0.61(0.19)
0.98(0.27)
0.93(0.36)
0.80(0.33)
1.12(1.76)
1.35(0.25)
0.83(0.20)
0.70(0.25)
DIEM
Our
GBVS [11]
PQFT [18]
Rudoy [17]
OBDL [14]
SALICON [22]
Xu [15]
BMS [10]
SalGAN [26]
AUC 0.86(0.08)
0.81(0.09)
0.71(0.11)
0.80(0.11)
0.75(0.14)
0.79(0.11)
0.80(0.11)
0.77(0.11)
0.81(0.08)
NSS
2.25(1.16)
1.21(0.82)
0.86(0.71)
1.40(0.83)
1.26(1.03)
1.68(1.04)
1.34(0.74)
1.20(0.80)
1.60(0.71)
CC
0.49(0.21)
0.30(0.18)
0.19(0.14)
0.38(0.20)
0.29(0.22)
0.36(0.19)
0.35(0.17)
0.28(0.17)
0.35(0.13)
KL
1.30(0.55)
1.64(0.48)
1.73(0.44)
2.33(2.05)
2.77(1.58)
1.66(0.58)
1.67(0.39)
1.96(1.13)
1.64(0.41)

KL divergence

1.35

1.3

1.25
1

0.75

0.5

0.25

Feature dropout rate pf

0

0

0.25

0.5

0.75

1

Hidden dropout rate ph

Fig. 13. The KL divergences of our models, using different drop out rates in
2C-LSTM.

that the human fixations more likely fall into the objects,
especially the moving objects or moving regions in the objects.
Additionally, we found that the correlation of attention across
consecutive frames is high. In light of these findings, the
OM-CNN architecture was proposed to explore the spatiotemporal features of object and motion to predict the intraframe saliency of videos, and the 2C-LSTM architecture was
developed to model the inter-frame saliency of videos. Finally,
the experimental results verified that our DNN based method
significantly outperforms other 8 state-of-the-art methods over
both our and other two public video eye-tracking databases,
in terms of AUC, CC, NSS, and KL metrics.
There are two promising directions for future work. First,
our method mainly focuses on videos with objects. Actually,
a handful of videos are of natural scenes, without any salient
object. Hence, saliency prediction of natural scene videos is
an interesting research direction in future. The second future
work is the potential application of our method in perceptual
video coding. In particular, our method is able to locate salient
and non-salient regions in videos, and it is expected that the
coding efficiency of videos can be improved by removing
the perceptual redundancy existing in the non-salient regions.
Consequently, the less bits are needed to encode and deliver
videos, greatly relieving the bandwidth-hungry issue in video
transmission.
R EFERENCES
[1] E. Matin, Saccadic suppression: a review and an analysis, Psychological
bulletin 81 (12) (1974) 899–917.

[2] A. Borji, L. Itti, State-of-the-art in visual attention modeling, Pattern
Analysis and Machine Intelligence, IEEE Transactions on 35 (1) (2013)
185–207.
[3] L. Itti, N. Dhavale, F. Pighin, Realistic avatar eye and head animation
using a neurobiological model of visual attention, Optical Science and
Technology 64 (2004) 64–78.
[4] Z. Ren, S. Gao, L.-T. Chia, D. Rajan, Regularized feature reconstruction
for spatio-temporal saliency detection, IEEE Transactions on Image
Processing 22 (8) (2013) 3120–3132.
[5] T. V. Nguyen, M. Xu, G. Gao, M. Kankanhalli, Q. Tian, S. Yan, Static
saliency vs. dynamic saliency: a comparative study, in: ACMM, ACM,
2013, pp. 987–996.
[6] S.-h. Zhong, Y. Liu, F. Ren, J. Zhang, T. Ren, Video saliency detection
via dynamic consistent spatio-temporal attention modelling, in: AAAI,
2013.
[7] S.-H. Lee, J.-H. Kim, K. P. Choi, J.-Y. Sim, C.-S. Kim, Video saliency
detection based on spatiotemporal feature learning, in: ICIP, 2014, pp.
1120–1124.
[8] T. Judd, K. Ehinger, F. Durand, A. Torralba, Learning to predict where
humans look, in: ICCV, pp. 2106–2113.
[9] M.-M. Cheng, N. J. Mitra, X. Huang, P. H. Torr, S.-M. Hu, Global
contrast based salient region detection, IEEE Transactions on Pattern
Analysis and Machine Intelligence 37 (3) (2015) 569–582.
[10] J. Zhang, S. Sclaroff, Exploiting surroundedness for saliency detection:
a boolean map approach, IEEE transactions on pattern analysis and
machine intelligence 38 (5) (2016) 889–902.
[11] J. Harel, C. Koch, P. Perona, Graph-based visual saliency, in: NIPS,
2006, pp. 545–552.
[12] F. Zhou, S. Bing Kang, M. F. Cohen, Time-mapping using space-time
saliency, in: CVPR, 2014, pp. 3358–3365.
[13] L. Zhang, M. H. Tong, G. W. Cottrell, Sunday: Saliency using natural
statistics for dynamic analysis of scenes, in: Annual Cognitive Science
Conference, 2009, pp. 2944–2949.
[14] S. Hossein Khatoonabadi, N. Vasconcelos, I. V. Bajic, Y. Shan, How
many bits does it take for a stimulus to be salient?, in: CVPR, 2015,
pp. 5501–5510.
[15] M. Xu, L. Jiang, X. Sun, Z. Ye, Z. Wang, Learning to detect video
saliency with hevc features, IEEE Transactions on Image Processing
26 (1) (2017) 369–385.
[16] L. Itti, P. Baldi, Bayesian surprise attracts human attention, Vision
research 49 (10) (2009) 1295–1306.
[17] D. Rudoy, D. B. Goldman, E. Shechtman, L. Zelnik-Manor, Learning
video saliency from human gaze using candidate selection, in: CVPR,
2013, pp. 1147–1154.
[18] C. Guo, L. Zhang, A novel multiresolution spatiotemporal saliency
detection model and its applications in image and video compression,
IEEE Transactions on Image Processing 19 (1) (2010) 185–198.
[19] Q. Zhang, Y. Wang, B. Li, Unsupervised video analysis based on a
spatiotemporal saliency detector, arXiv preprint arXiv:1503.06917.
[20] M. Kümmerer, L. Theis, M. Bethge, Deep gaze i: Boosting saliency
prediction with feature maps trained on imagenet, arXiv preprint
arXiv:1411.1045.
[21] S. S. Kruthiventi, K. Ayush, R. V. Babu, Deepfix: A fully convolutional
neural network for predicting human eye fixations, IEEE Transactions
on Image Processing.
[22] X. Huang, C. Shen, X. Boix, Q. Zhao, Salicon: Reducing the semantic
gap in saliency prediction by adapting deep neural networks, in: ICCV,
2015, pp. 262–270.

13

[23] J. Pan, K. McGuinness, E. Sayrol, N. O’Connor, X. Giro-i Nieto,
Shallow and deep convolutional networks for saliency prediction, arXiv
preprint arXiv:1603.00845.
[24] X. Li, L. Zhao, L. Wei, M.-H. Yang, F. Wu, Y. Zhuang, H. Ling, J. Wang,
Deepsaliency: Multi-task deep neural network model for salient object
detection, IEEE Transactions on Image Processing 25 (8) (2016) 3919–
3930.
[25] L. Wang, L. Wang, H. Lu, P. Zhang, X. Ruan, Saliency detection with
recurrent fully convolutional networks, in: ECCV.
[26] J. Pan, C. Canton, K. McGuinness, N. E. O’Connor, J. Torres, E. Sayrol,
X. a. Giro-i Nieto, Salgan: Visual saliency prediction with generative
adversarial networks, in: arXiv, 2017.
[27] Ç. Bak, A. Erdem, E. Erdem, Two-stream convolutional networks for
dynamic saliency prediction, arXiv preprint arXiv:1607.04730.
[28] L. Bazzani, H. Larochelle, L. Torresani, Recurrent mixture density network for spatiotemporal visual attention, arXiv preprint
arXiv:1603.08199.
[29] W. Wang, J. Shen, L. Shao, Video salient object detection via fully
convolutional networks, IEEE Transactions on Image Processing.
[30] S. Goferman, L. Zelnik-Manor, A. Tal, Context-aware saliency detection,
IEEE Transactions on Pattern Analysis and Machine Intelligence 34 (10)
(2012) 1915–1926.
[31] L. Itti, C. Koch, E. Niebur, A model of saliency-based visual attention
for rapid scene analysis, IEEE Transactions on Pattern Analysis and
Machine Intelligence 20 (11) (1998) 1254–1259.
[32] Y. Fang, W. Lin, Z. Chen, C.-M. Tsai, C.-W. Lin, A video saliency
detection model in compressed domain, IEEE transactions on circuits
and systems for video technology 24 (1) (2014) 27–38.
[33] J. Li, Y. Tian, T. Huang, W. Gao, Probabilistic multi-task learning for
visual saliency estimation in video, International Journal of Computer
Vision 90 (2) (2010) 150–165.
[34] K. Simonyan, A. Zisserman, Very deep convolutional networks for largescale image recognition, arXiv preprint arXiv:1409.1556.
[35] Y. Du, W. Wang, L. Wang, Hierarchical recurrent neural network for
skeleton based action recognition, in: CVPR, 2015, pp. 1110–1118.
[36] J. Redmon, S. Divvala, R. Girshick, A. Farhadi, You only look once:
Unified, real-time object detection, in: CVPR.
[37] M. Cornia, L. Baraldi, G. Serra, R. Cucchiara, A deep multi-level
network for saliency prediction, arXiv preprint arXiv:1609.01064.
[38] C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan,
V. Vanhoucke, A. Rabinovich, Going deeper with convolutions, in:
CVPR, 2015, pp. 1–9.
[39] S. Chaabouni, J. Benois-Pineau, O. Hadar, C. B. Amar, Deep learning for
saliency prediction in natural video, arXiv preprint arXiv:1604.08010.
[40] Y. Liu, S. Zhang, M. Xu, X. He, Predicting salient face in multiple-face
videos, in: CVPR, 2017.
[41] K. Holmqvist, M. Nystrm, R. Andersson, R. Dewhurst, H. Jarodzka,
J. Van de Weijer, Eye tracking: A comprehensive guide to methods and
measures, OUP Oxford, 2011.
[42] L. Itti, Automatic foveation for video compression using a neurobiological model of visual attention, IEEE Transactions on Image Processing
13 (10) (2004) 1304–1318.
[43] H. Hadizadeh, M. J. Enriquez, I. V. Bajic, Eye-tracking database for a
set of standard video sequences, IEEE Transactions on Image Processing
21 (2) (2012) 898–903.
[44] P. K. Mital, T. J. Smith, R. L. Hill, J. M. Henderson, Clustering of
gaze during dynamic scene viewing is predicted by motion, Cognitive
Computation 3 (1) (2011) 5–24.
[45] S. Mathe, C. Sminchisescu, Actions in the eye: dynamic gaze datasets
and learnt saliency models for visual recognition, IEEE transactions on
pattern analysis and machine intelligence 37 (7) (2015) 1408–1424.
[46] Y. Fang, Z. Wang, W. Lin, Z. Fang, Video saliency incorporating
spatiotemporal cues and uncertainty weighting, IEEE Transactions on
Image Processing 23 (9) (2014) 3910–3921.
[47] T. Mauthner, H. Possegger, G. Waltner, H. Bischof, Encoding based
saliency detection for videos and images, in: CVPR, 2015, pp. 2494–
2502.
[48] R. Carmi, L. Itti, Visual causes versus correlates of attentional selection
in dynamic scenes, Vision research 46 (26) (2006) 4333–4345.
[49] M. Marszalek, I. Laptev, C. Schmid, Actions in context, in: CVPR,
IEEE, 2009, pp. 2929–2936.
[50] M. Rodriguez, Spatio-temporal maximum average correlation height
templates in action recognition and video summarization.
[51] F. Boulos, W. Chen, B. Parrein, P. Le Callet, Region-of-interest intra
prediction for h. 264/avc error resilience, in: ICIP, IEEE, 2009, pp. 3109–
3112.

[52] Z. Li, S. Qin, L. Itti, Visual attention guided bit allocation in video
compression, Image and Vision Computing 29 (1) (2011) 1–14.
[53] M. Dorr, T. Martinetz, K. R. Gegenfurtner, E. Barth, Variability of eye
movements when viewing dynamic natural scenes, Journal of vision
10 (10) (2010) 28–28.
[54] N. Riche, M. Mancas, D. Culibrk, V. Crnojevic, B. Gosselin, T. Dutoit,
Dynamic saliency models and human attention: a comparative study on
videos, in: ACCV, Springer, 2012, pp. 586–598.
[55] A. Coutrot, N. Guyader, An efficient audiovisual saliency model to predict eye positions when looking at conversations, in: Signal Processing
Conference (EUSIPCO), 2015 23rd European, IEEE, 2015, pp. 1531–
1535.
[56] S. Marat, M. Guironnet, D. Pellerin, Video summarization using a visual
attention model, in: Signal Processing Conference, IEEE, 2007, pp.
1784–1788.
[57] Y. Gitman, M. Erofeev, D. Vatolin, B. Andrey, Semiautomatic visualattention modeling and its application to video compression, in: ICIP,
IEEE, 2014, pp. 1105–1109.
[58] H. Alers, J. A. Redi, I. Heynderickx, Examining the effect of task
on viewing behavior in videos using saliency maps, in: IS&T/SPIE
Electronic Imaging, International Society for Optics and Photonics,
2012, pp. 82910X–82910X.
[59] R. J. Peters, L. Itti, Beyond bottom-up: Incorporating task-dependent
influences into a computational model of spatial attention, in: CVPR,
IEEE, 2007, pp. 1–8.
[60] A. Coutrot, N. Guyader, Toward the introduction of auditory information
in dynamic visual attention models, in: 2013 14th International Workshop on Image Analysis for Multimedia Interactive Services (WIAMIS),
IEEE, 2013, pp. 1–4.
[61] T. T. INC., Tobii tx300 eye tracker, http://www.tobiipro.com/productlisting/tobii-pro-tx300/.
[62] U. Rajashekar, I. van der Linde, A. C. Bovik, L. K. Cormack, Gaffe:
A gaze-attentive fixation finding engine, IEEE transactions on image
processing 17 (4) (2008) 564–573.
[63] A. Dosovitskiy, P. Fischer, E. Ilg, P. Hausser, C. Hazirbas, V. Golkov,
P. van der Smagt, D. Cremers, T. Brox, Flownet: Learning optical flow
with convolutional networks, in: ICCV, 2015, pp. 2758–2766.
[64] S. Xingjian, Z. Chen, H. Wang, D.-Y. Yeung, W.-K. Wong, W.-c.
Woo, Convolutional lstm network: A machine learning approach for
precipitation nowcasting, in: NIPS, 2015, pp. 802–810.
[65] Y. Gal, Z. Ghahramani, A theoretically grounded application of dropout
in recurrent neural networks, in: NIPS, 2016, pp. 1019–1027.
[66] X. Glorot, Y. Bengio, Understanding the difficulty of training deep feedforward neural networks, in: Proceedings of the Thirteenth International
Conference on Artificial Intelligence and Statistics, 2010, pp. 249–256.
[67] D. Kingma, J. Ba, Adam: A method for stochastic optimization, arXiv
preprint arXiv:1412.6980.

