1

Unsupervised Object-Level Video Summarization
with Online Motion Auto-Encoder

arXiv:1801.00543v1 [] 2 Jan 2018

Yujia Zhang, Xiaodan Liang, Dingwen Zhang, Min Tan, and Eric P. Xing

Abstract—Unsupervised video summarization plays an important role on digesting, browsing, and searching the ever-growing
videos everyday. Despite the great progress achieved by prior
works (e.g., the frame-level video summarization), the underlying
fine-grained semantic and motion information (i.e., objects of
interest and their key motions) in online videos has been barely
touched, which is more essential and beneficial for many downstreaming tasks (e.g., object retrieval) in an intelligent system. In
this paper, we investigate a pioneer research direction towards
the fine-grained unsupervised object-level video summarization.
It can be distinguished from existing pipelines in two aspects:
extracting key motions of participated objects, and learning to
summarize in an unsupervised and online manner that is more
applicable for online growing videos. To achieve this goal, we
propose a novel online motion Auto-Encoder (online motion-AE)
framework that functions on the super-segmented object motion
clips. The online motion-AE mimics the online dictionary learning
for memorizing past states of object motions by continuously
updating a tailored recurrent auto-encoder network. This online
updating scheme enables the differentiable optimization of jointly
online feature learning and dictionary learning to discriminate
key object-motion clips. Finally, the key object-motion clips can
be mined by using the reconstruction errors obtained by the online motion-AE. Comprehensive experiments on a newly-collected
surveillance dataset and the public Base jumping, SumMe, and
TVSum datasets have demonstrated the effectiveness of online
motion-AE, and the application potential of fine-grained objectlevel video summarization.
Index Terms—Object-level video summarization, online motion
Auto-Encoder, stacked sparse LSTM auto-encoder.

I. I NTRODUCTION

V

IDEO has rapidly become one of the most common
sources of visual information. The amount of online
accessible video data is daunting with the success of mobile
phones, activity cameras, Google Glass, etc. Consequently, automatic tools for analyzing and understanding video contents
are essential for the large-scale intelligent system. In particular,
automatic video content summarization techniques, e.g., [1]–
[5], have received wide research interest in recent years due
to its huge application potentials. The goal is to compactly
depict the original video, distilling its important events into a
short watchable synopsis.
Y. Zhang and M. Tan are with Institute of Automation, Chinese Academy of
Sciences; School of Computer and Control Engineering, University of Chinese
Academy of Sciences, Beijing, China. (email: zhangyujia2014@ia.ac.cn,
min.tan@ia.ac.cn.) Work done while the first author was at CMU.
X. Liang and Eric P. Xing are with Machine Learning Department, Carnegie
Mellon University, Pittsburgh, PA, USA. (email:xdliang328@gmail.com, epxing@cs.cmu.edu.)
D. Zhang is with School of Automation, Northwestern Polytechnical
University, Xi’an, China. (e-mail: zdw2006yyy@mail.nwpu.edu.cn.)

Fig. 1. Different types of video summarization techniques. Specifically,
existing methods including (a) keyframe, (b) key subshot (c) key object-based
(only spatial) video summarization. Our proposed unsupervised object-level
(motion-clip based) video summarization technique (d) attempts to mine key
object motion clips to compactly depict the whole video.

As shown in Fig. 1 (a)−(c), the existing video summarization techniques often shorten the input video mainly in three
different ways, i.e., keyframe selection, key subshot selection,
key object selection within frames, respectively. Specifically,
the keyframe-based approaches (e.g. [4], [6]–[8]) generate the
video summary comprised by a collection of key frames, while
the key subshot-based approaches (e.g. [9]–[12]) select the
representative subshots of frames to form the video summary.
Recently, the key object-based approach [13] proposes to break
down the whole video into several single frames that reveal
the representative objects existing in a given video.
Despite the great progress achieved by the prior works, the
underlying fine-grained semantic and motion information (i.e.,
objects of interest and their key motions) in online videos has
been barely touched, which is more essential and beneficial
for many down-streaming tasks (e.g., object retrieval) in an
intelligent system. Besides, among the prior works, most of
them (e.g., [14]–[16]) address the ill-posed supervised or
semi-supervised video summarization with the requirement of
predefining the patterns of key frames, which is impractical
and not scalable to handle diverse and complicated evergrowing video contents.
To this end, this paper investigates a pioneer research
direction towards the fine-grained unsupervised object-level
video summarization as show in Fig. 1 (d) and Fig. 2.
Such new video summarization can be distinguished from the
existing ones mainly in two aspects: extracting key motions
of participated objects and learning to summarize in an unsupervised and online manner that is more applicable for online
growing videos. Compared with the conventional keyframebased and key subshot-based approaches, the investigated

2

Fig. 2. The proposed unsupervised object-level video summarization attempts
to mine key object motion clips to compactly depict the whole video. The
dashed lines in the figure indicate the moving trajectories of the object
instances in the extracted key object motion clips. Essentially, the video
summaries generated by the proposed approach can not only answer the
question of “what are the representative objects residing in the video?” but
also answer “what attractable actions of these objects are occurring in the
video?”

video summarization pipeline could realize more fine-grained
video summarization, focusing on the foreground moving
object instances rather than the entire image scenes. Compared
with the key object-based approach, our method takes the
temporal motion into consideration, which can better match the
nature of the video sequence. Essentially, the video summaries
generated by the proposed approach can not only answer the
question of “what are the representative objects residing in
the video?” but also answer “what attractable actions of these
objects are occurring in the video?”
To achieve the goal of the fine-grained object-level video
summarization, we propose a novel online motion AutoEncoder (online motion-AE) framework that functions on the
super-segmented object motion clips in an unsupervised way.
Online motion-AE mimics the online dictionary learning for
memorizing past states of object motions by continuously
updating a tailored recurrent auto-encoder network, that is,
a stacked sparse Long Short-Term Memory (LSTM) autoencoder. This online updating scheme enables the differentiable optimization of jointly online feature learning and
dictionary learning to discriminate key motion clips in an
unsupervised manner for online growing videos.
The concrete framework as well as a brief illustration are
shown in Fig. 3. More specifically, online motion-AE has
the following merits. Firstly, in contrast with the previous
supervised video summarization approaches like [14]–[16]
that rely on the video dataset with human-created summary
ground-truths for learning summarization models, the proposed approach does not require any annotations and yet
is capable of automatically mining representative patterns of
the object motion clips via an encoder-decoder mechanism.
Secondly, rather than following several manually designed
criteria to prioritize and select frames or subshots from videos
as prior unsupervised video summarization methods [2], [6],
[17] do, our model can learn an intrinsic measurement from the
unsupervised reconstruction errors of our model to judge the
distinctiveness of each object motion clip. Finally, compared
with the group sparse coding approach for online summarization [5], our online motion-AE model can achieve both

the online updating of the encoded motion representation for
each object motion clip and the adaptive adjustment of the
reconstruction state by a unified deep LSTM encoder-decoder
network.
In summary, this paper makes the following four-fold contributions:
• Key object motion-based video summarization. We explore a pioneer research direction towards the fine-grained
unsupervised video summarization that dives into the key
object motion clips of a video stream to compactly depict
the whole video and generate video summaries.
• Unsupervised online dictionary learning. We propose
a novel online motion-AE model, which can mimic the
online dictionary learning for memorizing past states
of object motions by continuously updating a tailored
recurrent auto-encoder network. Such online updating
scheme enables the differentiable optimization of the joint
online feature learning and dictionary learning and thus
can effectively discriminate the desired key object-motion
clips.
• The newly-collected OrangeVille benchmark. A new
surveillance video dataset is collected that allows for the
objective evaluation of our new field of video summarization methods. We provide the spatial-temporal annotations for all key object motion clips to push forward the
video summarization research with diverse granularities.
• State-of-the-art performance for both object motionlevel and frame-level summarization. Besides the key
object motion-based summarization comparison on OrangeVille, we also conduct comprehensive experiments
on other existing video summarization benchmarks (the
Base jumping, SumMe, and TVSum benchmark), and
compare with several state-of-arts to demonstrate the
effectiveness of the proposed approach.
II. R ELATED W ORK
There are mainly four lines of the existing video summarization techniques, i.e., the keyframe-based approach, the key
subshot-based approach, the key object-based approach, and
others, respectively.
The keyframe-based video summarization methods [3], [4],
[6], [7], [14], [17], [18] aim at identifying a series of discontinuous frames to form a summary that can represent the
main video content well. For example, Mundur et al. [7]
proposed to represent the video frame contents as multidimensional point data and use the Delaunay Triangulationbased clustering method to generate good quality summaries
with fewer frames and less redundancy. Khosla et al. [6] used
web-images as a prior to facilitate video summarization. The
intuition is that people tend to take pictures of objects and
events from a few canonical viewpoints in order to capture
them in a maximally informative way. Kim et al. [4] achieved
video summarization by diversity ranking on the similarity
graphs between Flickr images and YouTube video frames.
They formulated the storyline graphs as the inference of sparse
time-varying directed graphs from a set of photo streams with
assistance of videos.

3

Fig. 3. The framework of the proposed online motion-AE model. Briefly, it sequentially reads and preprocesses each sub video sequence in an online manner.
For each sub video sequence, we first track all the appearing object instances (as shown in (a)) and segment each of them into multiple object motion clips (as
shown in (b)). Then, the deep LSTM auto-encoder obtained previously is adopted to select the key object motion clips in the current video sequence. Next, all
the clips in the current video sequence are used to update the deep LSTM auto-encoder, which will be used in the next upcoming sub video sequence. Notice
that the dashed arrows indicate the motion trajectories of the object instance. Specifically, the yellow/green dashed arrows in (a) and (b) indicate the motion
trajectories before/after the current video frame shown in the figure. The blue ones in (c) indicate the motion trajectories of the summarized key motion clips.
The red dots in (a), (b), and (c) indicate the separation point of adjacent object motion clips.

Different from the keyframe-based approaches, the key
subshot-based video summarization approaches [9]–[11], [19],
[20] aim at identifying a series of defining subshots, each of
which is a temporally contiguous set of frames spanning a
short time interval. For example, Yao et al. [20] presented
highlight-driven based video summarization by exploring the
moments of user interests. Specifically, they decomposed video
segments into spatial and temporal streams to predict highlight
score, and single out video segments which have the maximum
sum of highlight scores. Song et al. [21] presented to use coarchetypal analysis technique to select shots which are most
relevant to canonical visual concepts shared between video and
images, according to the results of title-based image search.
Zhang et al. [22] achieved video summarization via sparse
coding with generalized sparse group lasso to learn a dictionary of video features and a dictionary of spatial-temporal
feature correlation graphs, and thus informative segments can
be generated. Laganire et al. [9] studied of spatial-temporal
activity within the video and proposed to construct the final
summary by selecting the clips showing the highest level of activity. Lu et al. [11] presented a video summarization approach
that can discover the story of an egocentric video, which
is based on a selection objective that captures the influence
between subshots as well as shot importance and diversity. The
method proposed by Zhang et al. [16] can implement video
summarization either on the frame level or the subshot level.
They designed a novel subset selection technique that can
transfer summary structures from annotated videos to unseen

testing videos. More recently, Meng et al. [13] proposed to
summarize the video content into a collection of key objects,
leading to the key object-based video summarization. Specifically, they formulated this representative selection problem as
a sparse dictionary selection problem, i.e., choosing a few representative object proposals to reconstruct the whole proposal
pool, then proposed to incorporate object proposal prior and
locality prior in the feature space when selecting representative
objects. The summarized key objects can be potentially used to
facilitate the understanding of the video content in object level.
Besides the aforementioned video summarization manners,
there are also some other interesting approaches, e.g, [23]–
[27]. For example, Sun et al. [24] proposed to summarize
video content by first finding the “montageable moments”
and then identifying salient people and actions to depict in
each montage. Pritch et al. [25] proposed to summarize an
endless video (usually generated by webcams or surveillance
cameras) by generating the corresponding video synopsis as
a response to the users query. Zhang et al. [26] and Chu et
al. [27] made attempt to summarize the related video content
from a given video collection rather than a single video. Such
methods can effectively mine the co-occurring spatial-temporal
patterns across multiple related videos.
III. O NLINE M OTION AUTO -E NCODER ( ONLINE
MOTION -AE)
The proposed online motion-AE framework resolves the
fine-grained unsupervised video summarization problem by

4

training a stacked sparse LSTM auto-encoder in an online
manner. In next sections, we first introduce the main stacked
sparse LSTM auto-encoder module, and then present the
details in building our online motion-AE framework.
A. The Stacked Sparse LSTM Auto-Encoder
The key towards the fine-grained unsupervised object-level
video summarization is the model capability of capturing
key object-level motion clips within each video. Inspired by
the success achieved by recurrent neural networks (especially
Long Shot-Term Memory (LSTM) [28]) on sequential modeling for images [29]–[31] and videos [32]–[35], our core
learning module is established by plugging the hierarchical
three LSTM layers into a generative auto-encoder model with
the sparse constraint, as illustrated in Fig. 4. The sparse LSTM
auto-encoder is thus able to learn the intrinsic motion patterns
to reconstruct the input object sequence with the least possible
amount of distortion. The learned parameters can be analogous
to a compact “motion dictionary” of past observed object clips.
Given a new object clip, we can easily judge its distinctness
with respect to past ones by the reconstruction error induced
from the learned LSTM auto-encoder model.
Specifically, the sparse LSTM auto-encoder model consists
of a LSTM encoder and a LSTM decoder. The LSTM encoder
recurrently takes each feature in the object clip as the input,
and feed-forward it into the hierarchal LSTM layers at each
time-step. After passing all features of the clip at all time-steps,
a motion context vector is generated to encode the holistic
temporal motion and appearance patterns. On the other hand,
the LSTM decoder network decodes the extracted motion
context vector by three symmetric LSTM layers to obtain
a synthesized representation sequence. The sparse constraint
is imposed on the encoded motion context vector to ensure
the compactness and generalization capability of “the motion
dictionary”. The reconstruction error between the synthesized
features and input features is used as a measurement to determine whether this input sample has already been memorized
by the recurrent auto-encoder. The unsupervised optimization
of the sparse LSTM auto-encoder can be realized by the differential back-propagation algorithm to minimize the difference
between the reconstructed features and input features. In the
next, we will go through some details of the basic LSTM unit,
the LSTM auto-encoder, the sparse LSTM auto-encoder, and
the stacked sparse LSTM auto-encoder in order.
LSTM Unit. As shown in Fig. 4 (A), the basic LSTM unit,
which is used for long-sequence modeling, applies a memory
cell ct to record and encode the history of the knowledge of
the inputs observed in previous time steps and determine the
hidden cell in the current time step ht . Here, the memory
cell ct and hidden cell ht are modulated by nonlinear gate
functions (e.g., the forget gate, output gate, and input gate),
which determine whether the LSTM keeps the values at the
gates (if the gates evaluate to 1) or discard them (if the gates
evaluate to 0) [36].
Specifically, there are three types of gates: the input gate
(i) controls whether the LSTM considers its current input xt ,
the forget gate (f) allows the LSTM to forget its previous

memory ct , and the output gate (o) decides how much of
the memory to transfer to the hidden states ht . These gates
together lead to the ability on learning complex long-sequence
modeling for LSTM. In particular, the forget gate (f) serves
as a time-varying data-dependent on/off switch to selectively
incorporating the past and present knowledge [37].
The concrete formulations are defined as:
it = σ(Wix xt + Φih ht−1 + bi ),
ft = σ(Wf x xt + Φf h ht−1 + bf ),
ot = σ(Wox xt + Φoh ht−1 + bo ),

(1)

ct = it ⊗ φ(Wcx xt + Φch ht−1 + bc ) + ft ⊗ ct−1 ,
ht = ot ⊗ φ(ct ),
where σ is a sigmoid function, φ is the hyperbolic tangent function tanh, ⊗ donates element-wise product, W =
{Wix , Wf x , Wox , Wcx } is the transform from the input to
LSTM states, Φ = {Φih , Φf h , Φoh , Φch } is the recurrent
transformation matrix between the hidden states, and b is the
bias vector.
LSTM Auto-Encoder. In LSTM auto-encoder, both the
encoder network and the decoder network are built upon the
LSTM units. Given an input object clip X = (x1 , · · · , xT ), the
LSTM encoder recurrently output hidden states (h1 , · · · , hT )
with shared network parameters ΘE = {WE , ΦE , bE }, where
WE is the transform from the input to LSTM encoder states.
ΦE is the recurrent transformation matrix between the encoder
states, and bE is the encoder bias vector. Symmetrically,
the LSTM decoder recurrently decodes hidden states with
the shared network parameters ΘD = {WD , ΦD , bD }, and
generates the current reconstruction output Y = (y1 , · · · , yT )
via an additional linear mapping:
yt = Wyh h0t + bh ,

(2)

where {Wyh , bh } are the reconstruction mapping parameters,
and h0t is the hidden state inferred for reconstructing the t-th
input feature.
Training the whole LSTM auto-encoder is to optimize
the parameters {ΘE , ΘD , Wyh , bh } by minimizing the meansquared reconstruction error between the input sequence and
the corresponding reconstructed sequence via:
L(X, Y) =

arg min
ΘE ,ΘD ,Wyh ,bh

T
1 X
||xt − yt ||22 .
2T t=1

(3)

Sparse LSTM Auto-Encoder. Motivated by the physiological evidence that describing patterns with less active neurons
minimizes the probability of destructive cross-talk [38], a
regularization term is applied to constraining the sparsity to
the target activation function [39], [40]. It penalizes a deviation
of the expected activation of the hidden states representation
from a fixed (low) level ρ by optimizing:
arg min

K
X

ΘE ,ΘD ,Wyh ,bh k=1

L(X

(k)

,Y

(k)

)+β

D
X

KL(ρ||b
ρd ),

d=1

ρ
(1 − ρ)
KL(ρ||b
ρd ) = ρ log
+ (1 − ρ) log
,
ρbd
(1 − ρbd )

(4)

5

Fig. 4. The illustration of (A) the LSTM unit and (B) the sparse LSTM auto-encoder (B). LSTM unit is a type of recurrent neural network, which models
long-range dependencies. Sparse LSTM auto-encoder is established by plugging the hierarchical three LSTM layers into a generative auto-encoder model with
the sparse constraint.

where X(k) indicates the k-th training sample in the training
set of totally K training sequences, D indicates the dimension
of the hidden states, ρ is the target average activation of each
PK
(k)
dimension of the hidden state hT , and ρbd = k=1 [hT ]d /K is
the average activation of the d-th dimension of the hidden state
[hT ]d over K training samples. The Kullback-Leibler (KL)
divergence KL(·) is used to impose the sparse constraint. As
claimed in sparse coding [39], a non-redundant over-complete
feature set can be learned when ρ is small. Here, we set ρ =
0.05 as suggested in [39].
The architecture of a sparse LSTM auto-encoder network
is shown in Fig. 4 (B), which is established by plugging
the hierarchical three LSTM layers into a generative autoencoder model with sparse constraint. Notice that the output
reconstruction sequence are in reverse order as compared with
the input sequence as suggested by [32], which could make the
optimization easier because the model can get off the ground
by looking at low range correlations.
Stacked Sparse LSTM Auto-Encoder. The final deep
network is established by stacking M = 3 sparse LSTM autoencoder layers as shown in Fig. 3 (d). To achieve the hierarchical latent representation, the hidden states (h1 , · · · , hT )
from the first (bottom) LSTM auto-encoder layer is posed
as the input (x1 , · · · , xT ) of the second LSTM auto-encoder
layer. The dimension of hidden states in each LSTM layer is
gradually decrease to capture more high-level motion representation by stacking more LSTM auto-encoder layers. Like many
stacked auto-encoder models (e.g., [32], [40]), the proposed
deep LSTM auto-encoder model can be easily trained (both
in the offline training stage and the online updating stage)
using greedy optimization: training each of the sparse LSTM
auto-encoder layer one by one via optimizing the objective
function (4) using stochastic gradient descent.
B. Online Video Summarization
Aiming at the fine-grained video summarization that dives
into object instances, our online learning framework incorporates the multiple object tracking and motion trajectory
segmentation to extract candidate object motion clips, and
then processes these clips with the stacked sparse LSTM autoencoder model in an online manner.
1) Preprocessing: Multiple Object Tracking. We first
track all moving objects in the video to discover all objects
of interest to be summarized. Notice that the single object

Fig. 5. An example of the motion trajectory segmentation. The motion
trajectory of each object instance was super-segmented into multiple object
motion clips, reflecting different motion status with unnoticeable changes.

tracking (SOT) techniques as used in some previous video
understanding systems [41]–[43] cannot be readily used in our
task as 1) they would miss many objects that are absence at
the first frame of the video (or shot), and 2) they tend to be
time-consuming when there are intensive objects appearing in
the video (e.g., the surveillance video in our case).
Due to arbitrary number of objects that may appear
in each frame, we adopt a state-of-the-art multiple object tracking algorithm, i.e., the Markov Decision Process
(MDP) tracker [44], which is a tracking-by-detection algorithm equipped with effective tracking model in handling the
appearance/disappearance of any object instance in the video.
The lifetime of each object is modeled with a MDP describing the transitions between four subspaces of states (active,
tracked, inactive, lost) which corresponds to seven actions.
All deterministic actions are learned from reward functions
to learn a policy which can maximize the total obtained
rewards. Data association is learned by a similarity function
which is as part of the MDP policy with reinforcement
learning. In this way, it can benefit from the advantages of
both offline and online learning. Particularly, the localization
of moving objects is achieved by performing the pre-trained
Faster RCNN [45] model on PASCAL VOC [46] dataset on
down-sampled frames.
Motion Trajectory Segmentation. With the goal of finding
the distinct motion patterns of each object, we thus supersegment the motion trajectory of each object instance into
multiple object motion clips, each of which reflects a certain motion status with unnoticeable changes as shown in
Fig. 5. One straight-forward way would be to cut each motion
trajectory into clips within the fixed length. However, such

6

Fig. 6. Context-aware feature representation. The orange boxes are the bounding box locations feeding into the Faster RCNN to extract the corresponding
features. (b) and (c) are global and local context-aware feature representation respectively.

arbitrarily cutting clips would destroy some continuous motion
sequences. An alternative way is to use the shot detection
approach [47], which is mainly based on changes in the color
histogram. This still cannot work well in our case as the
sequences formed by the cropped object trajectories usually
contain the image regions with similar color distribution (see
Fig. 5).
We thus adopt the recent superframe segmentation
method [1], which cuts video sequences mainly based on the
motion information by defining an energy function E(Sj ) [1]
for the measurement of a superframe Sj :
1
Pl (|Sj |),
1 + γCcut (Sj )
Ccut (Sj ) = min (Sj ) + mout (Sj ),
E(Sj ) =

(5)

where Ccut is the cut cost, γ controls the influence between
cost and length prior Pl , and min and mout are motion magnitude estimated in the first and last frame of each superframe
respectively. Each long motion trajectory for each object is
thus effectively split into several continuous motion clips with
variable lengths.
Context-aware Feature Representation. Given the supersegmented motion clips, we extract the context-aware feature
representation for each region as the inputs to sparse LSTM
auto-encoder. Specifically, we consider a key object motion
clip from two prospectives: the distinct object appearance
and distinguished surrounding context. We thus represent
each region with a 8192-dimension vector by concatenating
the appearance feature and the context feature extracted by
the Faster RCNN network [48]. Specifically, the appearance
feature is the 4096-dimension feature vector generated by
feeding the original object region into Faster RCNN network
and using the “fc7” layer pooled from the corresponding object
bounding box location (see Fig. 6 (a)). The context feature is
another 4096-dimension feature vector generated by feeding
the masked image, which is produced by replacing the pixel
values out of the expanded bounding box1 region with the
fixed mean pixel values pre-computed on ILSVRC 2012 [49]
into Faster RCNN network, and then using the “fc7” layer
pooled from the entire image scene (see Fig. 6 (b)). This
context-aware representation can not only capture the detailed
1 The

bounding box is expanded to 2 times of its original length and width.

appearance of each moving object, but also its corresponding
location and surrounding context.
2) Offline Training Process: The sparse LSTM autoencoder model is first initialized with an offline learning
process to obtain the basic reconstruction capability, which
is then updated in an online way. Specifically, we first collect
a 40-minute-long (with 83278 frames) video sequence as the
offline training data. Then, we randomly extract 30 bounding
boxes from each offline video frame and copy each bounding
box three times to form a still object sequence. In this way, the
offline learned model will have the knowledge information for
still object sequence, and thus can better differentiate moving
object clips. Afterwards, the context-aware feature of each still
sequence are thus used to train the stacked sparse LSTM autoencoder.
3) Online Summarization Process: Given each test video
sequence, the proposed online motion-AE sequentially reads
the video 1000 frames by 1000 frames, (i.e., each of the 1000frame sequence is treated as the sub video sequence as shown
in Fig. 3) and summarizes the key object motion clips. At
each step, we first extract candidate object motion clips, and
sample three object motion regions (i.e., the start, middle,
and end regions) from the clip as the input to the stacked
sparse LSTM auto-encoder. This down-sampling operation
help significantly reduces the computational complexity and
yet retains the informative temporal information. We then
extract the context-aware feature representation (as introduced
in III-B1) of the sampled object motion regions and use the
pre-trained stacked sparse LSTM auto-encoder to obtain the
reconstruction error of each object clip. Here the reconstruction errors can be naturally used as the summarization scores
for unsupervised video summarization. Intuitively, key object
motion clips with unseen appearance and motion cannot be
well reconstructed from the patterns learned by either the
offline video sequence or the previous video content in the
online test video. Next, we use all clips in the current sub video
sequence to update the stacked sparse LSTM auto-encoder
for capturing the underlying motion patterns, and the updated
network is employed on the upcoming sub video sequence. By
performing the aforementioned online process for the whole
video sequence, we can obtain the summarization score for
each object motion clip, and thus get the final results for key
object motion clip summaries.

7

IV. E XPERIMENT
A. Experimental Settings
Datasets. We evaluate the performance of the proposed
online motion-AE model on several datasets. The first one
is the newly collected video surveillance dataset named as
OrangeVille, which contains 30 surveillance video sequences
downloaded from the YouTube website with the keyword of
“OrangeVille” and filtering out the videos with low resolution and strong shadows. Each of the video in OrangeVille
dataset averagely contains 3000 frames with the length of
100 seconds. The videos are recorded in outdoor scenes
with moving objects mainly in categories of human, cars,
buses, bicycles, and motorcycles. The motion/action of such
objects are mainly waiting, slow moving, fast moving, and
turning around. Targeting at the fine-grained object-level video
summarization task that aims to discover all key object motion
clips, we provide the spatial-temporal object bounding-box
annotations to depict each key object motion clip in each
video on OrangeVille dataset. The ground-truth annotations
are labeled by a subject manually, focusing on the key object
motions like fast moving cars, pedestrian crossing the road,
and buses turning left/right.
The second dataset is the Base jumping dataset from public
CoSum dataset [27]. Since CoSum dataset is applied for video
co-summarization task, and the dataset is consisted of multiple
videos organized into groups with topic keywords, we use one
class of the whole dataset, i.e., the Base jumping class, to
evaluate our model.
Besides, we also use other two frame-level datasets, which
are the public SumMe dataset [1] and TVSum dataset [21].
Specifically, SumMe contains 25 videos covering holidays,
events and sports, such as Statue of Liberty, saving dolphins
and bike polo. TVSum contains 50 videos in 10 categories
downloaded from YouTube defined in the TRECVid Multimedia Event Detection (MED). The contents in these two
datasets are from both first-person and third-person videos,
which addresses complex image scenes.
Evaluation Metrics. We adopted three common metrics,
including the average precision (AP) score, the F-measure
score, and the AUC score, which have been used in previous
works (e.g., [15], [16] ). The AP score were used to evaluate
the generated summarization scores, which was generated by
binarizing the samples via varying the quantification threshold
within the range of [0, 1]. The resulting true positive rate (or
the recall rate) versus precision rate at each threshold value
formed the PR curve. Then the area under the PR curve was
calculated as the AP score. Similarly, the AUC score was
obtained by calculating the area under the ROC curve, which
was generated based on the false positive rate and the true
positive rate. Specifically, The precision P RE(P recision),
true positive rate T P R(Recall) and false positive rate F P R
were defined as

where SF , GF and GB denote the set of samples predicted
as 1, the samples labeled as 1, and the samples labeled as
0, respectively. Different from AUC and AP, F-measure was
adopted to evaluate the generated video summaries, which was
defined as:
2P recision × Recall
.
(7)
F =
P recision + Recall
During the evaluation prcoess, the prediction results of the
proposed approach as well as other compared state-of-theart methods for the fine-grained object-level/frame-level video
summarization tasks are the clip-level/frame-level real-valued
scores ranging from 0 to 1 (after normalization). The AP
scores and AUC scores can then be used to evaluate such
real-valued prediction results directly. The F-measure scores
were obtained by binarizing the predictions via a threshold
τ = 0.5.
In the fine-grained object-level video summarization task,
the ground-truth are G human labeled object motion clips
s
e
{fg , bsg , beg }G
g=1 , where fg ,bg , and bg indicate the temporal
location (i.e., the frame numbers) of the g-th ground-truth clip,
the spatial location (i.e., the object bounding box) in the start
of the frame, and the spatial location in the end of the frame,
respectively. Given an object motion clip {fp , bsp , bep } which
is predicted by the summarization algorithm, we calculate the
IOU overlaps both for the spatial locations and the temporal
locations as:
fp ∩ fg
,
IOU tem =
fp ∪ fg
(8)
s
s
bep ∩ beg
1 bp ∩ bg
IOU spa = ( s
s + e
e ).
2 bp ∪ bg
gp ∪ bg
Then, the object motion clips would be considered as correct
summarization if IOU tem and IOU spa are simultaneously
larger than the threshold ϕ. Due to the limited accuracy of
the object tracking and superframe segmentation algorithms,
we set ϕ = 0.1 during our evaluation.
In the frame-level video summarization task, the groundtruths are the binary labels for each superframe provided
by the dataset, while the prediction results are the predicted
summarization scores for those superframes. In this case, all
the evaluation metrics can be easily calculated because the
temporal locations of all the test superframes are finely aligned
with the corresponding ground-truths.
Implementation Details. We implemented our model on
the public TensorFlow platform. The dimensions of hidden
states of the first, second, and third sparse LSTM auto-encoder
layers were 4096, 2048, and 1024, respectively. In the offline
pre-training phase, we set the batch size as 100, while in
the online updating phase, the batch size was dynamically
determined by the number of clips extracted in each sub video
sequence. The learning rate was set to 0.001 during the offline
pre-training while 0.0001 during the online updating.
B. Fine-Grained Video Summarization

P RE = |SF ∩ GF |/|SF |,
T P R = |SF ∩ GF |/|GF |,
F P R = |SF ∩ GB|/|GB|,

(6)

In this section, we conducted extensive evaluation on finegrained object-level video summarization task on OrangeVille
dataset.

8

TABLE I
C OMPARISON OF THE FINE - GRAINED VIDEO SUMMARIZATION RESULTS BETWEEN THE PROPOSED APPROACH AND OTHER BASELINES ON
THE O RANGE V ILLE DATASET IN TERMS OF AUC SCORE , AP SCORE , AND F- MEASURE ( HIGHER VALUES INDICATE BETTER RESULTS ).

AUC score
AP score
F-measure

Sparse [5]
Coding
0.4252
0.1542
0.1284

Stacked Sparse Auto-Encoder [39]
1 layer
2 layers
3 layers
0.4512
0.4040
0.4354
0.1615
0.1493
0.1705
0.1325
0.1420
0.1662

Stacked LSTM Auto-Encoder [32]
1 layer
2 layers
3 layers
0.4829
0.5413
0.5680
0.2177
0.2476
0.2638
0.2310
0.2546
0.2795

Stacked Sparse LSTM Auto-Encoder(ours)
1 layer
2 layers
3 layers
0.4970
0.5598
0.5908
0.2275
0.2796
0.2850
0.2364
0.2721
0.2901

TABLE II
C OMPARISON OF DIFFERENT MASKS WHICH IS USED TO EXTRACT CONTEXT FEATURES ON STACKED SPARSE LSTM AUTO - ENCODER MODELS IN TERMS
OF AUC SCORE , AP SCORE , AND F- MEASURE ( HIGHER VALUES INDICATE BETTER RESULTS ).

AUC score
AP score
F-measure

1 layer
0.4706
0.2008
0.1841

mask = zero
2 layers
3 layers
0.5178
0.5775
0.2166
0.2420
0.1886
0.1996

1 layer
0.4970
0.2275
0.2364

mask = mean
2 layers
3 layers
0.5598
0.5908
0.2796
0.2850
0.2721
0.2901

TABLE III
C OMPARISON OF DIFFERENT FEATURES ON STACKED SPARSE LSTM AUTO - ENCODER MODELS IN TERMS OF AUC SCORE , AP SCORE , AND F- MEASURE
( HIGHER VALUES INDICATE BETTER RESULTS ).

AUC score
AP score
F-measure

1 layer
0.4889
0.1928
0.1839

global + local
2 layers
3 layers
0.4712
0.5175
0.1934
0.2026
0.1764
0.1362

1 layer
0.4628
0.1677
0.1050

object + global
2 layers
3 layers
0.4857
0.5018
0.1834
0.1864
0.1216
0.1404

1 layer
0.4970
0.2275
0.2364

object + local
2 layers
3 layers
0.5598
0.5908
0.2796
0.2850
0.2721
0.2901

TABLE IV
C OMPARISON OF THE NUMBER OF ONLINE UPDATES ON STACKED SPARSE LSTM AUTO - ENCODER MODELS IN TERMS OF AUC SCORE , AP SCORE , AND
F- MEASURE ( HIGHER VALUES INDICATE BETTER RESULTS ).

AUC score
AP score
F-measure

1 online update
1 layer
2 layers
3 layers
0.4427
0.5359
0.5826
0.2084
0.2577
0.2801
0.2228
0.2408
0.2684

2 online updates
1 layer
2 layers
3 layers
0.4970
0.5598
0.5908
0.2275
0.2796
0.2850
0.2364
0.2721
0.2901

Comparison with state-of-the-art and baseline learning
models. To demonstrate the superiority of the proposed online
motion-AE, we conducted comparisons with two state-ofarts of the learning models, which are the online sparse
coding as used in [5] and the stacked sparse auto-encoder
as introduced in [39], respectively. Particularly, the online
sparse coding proposed in [5] is one of the current stateof-the-art model for frame-level video summarization both
in the online and unsupervised fashion. Other state-of-the-art
video summarization approaches cannot be readily used in our
comparison as they do not work in the unsupervised and online
manner.
As for the online sparse coding-based approach, we firstly
used the extracted object motion clips in the offline video sequence to learn a dictionary. Then, in the online testing phase,
we used the previously constructed dictionary to calculate the
reconstruction error for each new object motion clip in the
current sub video sequence. We adopted the same motiontrajectory segmentation and context-aware feature representation modules used in our model to enable a fair comparison of
different learning approaches. As for the state-of-the-art model
of the stacked sparse auto-encoder, it can be implemented by

3 online updates
1 layer
2 layers
3 layers
0.5268
0.5491
0.5710
0.2369
0.2695
0.2630
0.2414
0.2539
0.2157

simply replacing the LSTM units with a single non-linear layer
in our model.
The qualitative evaluation results are shown in Fig. 7, from
which we can observe that the proposed online motion-AE
can effectively summarize the key object motion clips, e.g.,
the fast driving cars, the occasionally appeared walkers who
are crossing the road and turning around in the surveillance
video sequence. Whereas the stacked sparse auto-encoder and
sparse coding-based approaches are more likely to select nonkey object motion clips, e.g., the stationary cars waiting for
the traffic lights.
The quantitative evaluation results are shown in Table I. We
can observe that the auto-encoder based models can significantly outperform the online sparse coding methods, even with
a single layer, which demonstrates the good reconstruction
capability of auto-encoders.
We also made comparisons with several model variants
to show the effectiveness of each model component: 1) the
stacked LSTM auto-encoder that eliminates the sparsity on
the hidden states in each layer; 2) the shallow LSTM autoencoder with or without using the sparse constraint, which
only adopts one single LSTM auto-encoder layer to generate

9

TABLE V
C OMPARISON OF THE NUMBER OF LAYERS ON STACKED SPARSE LSTM AUTO - ENCODER MODELS IN TERMS OF AUC SCORE , AP SCORE , AND
F- MEASURE ( HIGHER VALUES INDICATE BETTER RESULTS ).

AUC score
AP score
F-measure

1 layer
0.4970
0.2275
0.2364

2 layers
0.5598
0.2796
0.2721

3 layers
0.5908
0.2850
0.2901

4 layers
0.5919
0.2582
0.2712

5 layers
0.5875
0.2719
0.2612

Fig. 7. Some examples of the fine-grained summarization results on the OrangeVille dataset. This shows the comparisons of our online motion-AE model
with sparse coding [5] and stacked sparse auto-encoder [39] models.

reconstruction errors; 3) the two-layer LSTM auto-encoder
models with or without the sparse constraint. From the obtained experimental results, it can be observed that using
hierarchical LSTM auto-encoder structure as our full model
can achieve better performance than the other variants that
only adopt one or two layers. This is due to the more
powerful hierarchical feature representations learned by our
model, which can benefit for capturing complex object motion
patterns.
Ablation analysis. We discuss different components in
our stacked sparse LSTM auto-encoder model including the
mask for extracting context feature, different types of feature
representation, the number of online updates during testing
phase and the number of layers for the stacked sparse LSTM
auto-encoder model:
1) the mask we used to extract context features, one is
produced by replacing the pixel values out of the expanded
bounding box region with the fixed mean pixel values which
are pre-computed on ILSVRC 2012 [48], and the other uses
value of zero to replace the original pixel values. As shown
in Table II, we can see that using mask of mean pixel value
performs better than mask of zero value in all layers, since the
settings for testing is the same as the one for training when
using the mask of the fixed mean pixel value;
2) the different types of features we used for object motion
clip feature representation, including the concatenating the
global context and local context features (global+local), the

appearance feature and global context feature (object+global),
and the appearance feature and local context feature (object+local). The way we concatenated feature is to just link the
two 4096-dimensional feature vectors together to get the final
8192-dimensional feature vector. The global and local context
feature are illustrated in Fig. 6 (b)(c). The results are shown
in Table III, and we can observe that local context-aware
feature have better representation ability, due to the reason
that local context preserves and highlights the moving regions
while global context would weaken motion information for the
moving objects, and combining two different types of features
also enhances the representative abilities of the model;
3) the number of online updates we used to update the
stacked sparse LSTM auto-encoder model during testing period. We test our model by replacing different layers of sparse
LSTM auto-encoder. From Table IV, we can see that using 1
and 2 online updates, the performance increases as the layers
increase, but the performance decreases for 3 online updates
due to the reason of over-learning;
4) the number of layers we used in our model. The results
in Table V demonstrate that the stacked sparse LSTM autoencoder often converges in the 3-rd layer and then would
slightly decline with 4, 5 and more layers, so we use 3layer stacked sparse LSTM auto-encoder model to learn the
representation of features on object motion clips and compute
reconstruction error for video summaries.

10

Fig. 8. Some examples of the frame-level summarization results on the Base jumping dataset. This shows the comparisons of our online motion-AE model
with sparse coding [5] and stacked sparse auto-encoder [39] models.

C. Frame-level Video Summarization
We demonstrate the generalization capability of our model
on frame-level video summarization task, which is evaluated
on the Base jumping, SumMe and TVSum datasets.
Our tailored pipeline for frame-level summarization on
Base jumping dataset. We adapt our model for frame-level
summarization, where the summarization scores of a certain
frame was obtained by averaging the reconstruction errors of
all object motion clips residing in it. In this way, the proposed
approach as well as the baseline methods can be conveniently
extended to obtain the frame-level summarization results.
The qualitative evaluation results are shown in Fig. 8, from
which we can see that the proposed online motion-AE can
summarize the key superframes that contain representative
object motion, e.g., the moment for the jumping and flying.
Whereas the stacked sparse auto-encoder and sparse codingbased approaches are more likely to select superframes with
less informative object motion.
Our tailored pipeline for frame-level summarization on
SumMe and TVSum datasets. We adapt our model for framelevel summarization, where the summarization scores of a
certain superframe was obtained first by adopting the recent
superframe segmentation method [1] to super-segment each
video into multiple superframes, which cuts video sequences
mainly based on the motion information. Then we used pretrained ResNet152 [50] model on ILSVRC 2015 [51] to extract
superframe motion features. After that, we used our online
motion-AE model to obtain the reconstruction errors of all
frame-level motion clips. In this way, the proposed approach
as well as the baseline methods can be conveniently extended
to obtain the frame-level summarization results. From the
summarization results of SumMe and TVSum datasets in
Fig. 9 and Fig. 10, we can see that the our online motion-AE
model can effectively capture the informative and temporal
motion patterns and thus obtain good results.

TABLE VI
C OMPARISON OF THE FRAME - LEVEL VIDEO SUMMARIZATION RESULTS
BETWEEN THE PROPOSED APPROACH AND OTHER Unsupervised
BASELINES ON S UM M E DATASET IN TERMS OF AUC SCORE , AP SCORE ,
AND F- MEASURE ( HIGHER VALUES INDICATE BETTER RESULTS ).
Method
Video MMR [52]
TVSum [21]
VSUMM1 [53]
VSUMM2 [53]
stacked GRU Auto-Encoder [54]
Online Motion-AE

F measure
0.266
0.266
0.328
0.337
0.354
0.377

TABLE VII
C OMPARISON OF THE FRAME - LEVEL VIDEO SUMMARIZATION RESULTS
BETWEEN THE PROPOSED APPROACH AND OTHER Unsupervised
BASELINES ON TVS UM DATASET IN TERMS OF AUC SCORE , AP SCORE ,
AND F- MEASURE ( HIGHER VALUES INDICATE BETTER RESULTS ).
Method
Web Image Prior [6]
LiveLight [5]
TVSum [21]
stacked GRU Auto-Encoder [54]
Online Motion-AE

F measure
0.360
0.460
0.500
0.510
0.515

Comparison with competitive learning models. The quantitative evaluation results for SumMe and TVSum datasets are
shown in Table VI and Table VII respectively. The state-of-arts
we choose are all unsupervised methods, which are the same
of our experimental settings for fair comparisons. As can be
seen, the proposed stacked sparse LSTM auto-encoder again
obtains better performance than other unsupervised baselines
on two different datasets. This demonstrates that the proposed
learning model can effectively capture the informative temporal motion patterns and thus obtain the outperforming performance. The performance gaps among different approaches
become smaller as compared with those in the fine-grained

11

Fig. 9. Some examples of the frame-level summarization results on the SumMe dataset using our online motion-AE model. This shows some summary results
from three videos in the dataset.

Fig. 10. Some examples of the frame-level summarization results on the TVSum dataset using our online motion-AE model. This shows some summary
results from three videos in the dataset.

video summarization task. The reason is that the fine-grained
video summarization requires the model be able to capture
detailed instance-level motion patterns, which is much more
challenging than the frame-level summarization task.
Underlying connections between the two tasks. Compared
with the key object-motion clips summarized by the finegrained object-level video summarization, the key frames
summarized by the frame-level video summarization consider
not only the distinct foreground object motions but also the attractable whole image scenes. Thus, generating the frame-level

summaries by only mining distinct object motion may partially
cover the desired summarization result, which explains the
performance gaps drop on the SumMe and TVSum datasets.
We can observe that the performance drop of the proposed
online motion-AE is not quite significant, which indicates that
key object motions may actually occupy a relatively large
portion of the factors in selecting frame-level summaries.
Consequently, this demonstrates the potential usage of the finegrained object-level video summarization approach.

12

ACKNOWLEDGMENT
This project is supported by the Department of Defense
under Contract No. FA8702-15-D-0002 with Carnegie Mellon
University for the operation of the Software Engineering
Institute, a federally funded research and development center.
R EFERENCES
Fig. 11. The boxplot of the user study results on the collected OrangeVille
dataset and the public Base jumping dataset on our online motion-AE model,
stacked sparse auto-encoder [39] and sparse coding [5] models.

D. User Study
We further provide human evaluation results on OrangeVille and Base jumping datasets to better verify the
model capability. Similar with [6], 15 subjects are asked to
watch the randomly selected videos at 3x speed and then
showed them the corresponding video summaries generated
by different approaches. They were asked to rate the overall
quality of each summarization result by assigning a rating
from 1 to 10 (higher is better). The user study results on
the OrangeVille dataset are reported in Fig. 11. Consistent
with the objective evaluation results obtained by the evaluation
metrics, our approach can consistently outperforms the stacked
sparse auto-encoder and spare coding models on the finegrained object-level video summarization task. In addition,
from the user study results, it can be seen that the proposed
model can obtain more consistent subjective evaluation scores
across different subject judges than the competitive models.
Thus, this experiment further verifies the effectiveness of the
proposed online motion-AE model.
V. C ONCLUSION
In this paper, we investigated a pioneer research direction towards the unsupervised object-level video summarization. By
diving into the fine-grained moving object instances residing
in each given video sequence, the proposed online motion-AE
approach can learn to summarize the extracted object motion
clips in an unsupervised and online manner. Specifically, the
online motion-AE can mimic the online dictionary learning
for memorizing past states of object motions by continuously
updating a tailored recurrent auto-encoder network, which
enables the jointly online feature learning and dictionary
learning to discriminate key motion clips. Comprehensive
experiments on a newly collected OrangeVille dataset as well
as public Base jumping, SumMe and TVSum datasets have
demonstrated the effectiveness of the proposed approach both
in the fine-grained video summarization and its application
potential in the frame-level video summarization.
For future work, we plan to introduce the computational
attention schemes [55]–[57] to improve the video summarization performance. On the other hand, we will apply the video
summarization technique for assisting some high-level video
understanding tasks like video object detection [58] and video
prediction [59].

[1] M. Gygli, H. Grabner, H. Riemenschneider, and L. Van Gool, “Creating
summaries from user videos,” in European conference on computer
vision, 2014, pp. 505–520.
[2] D. Potapov, M. Douze, Z. Harchaoui, and C. Schmid, “Category-specific
video summarization,” in European conference on computer vision,
2014, pp. 540–555.
[3] Y. J. Lee, J. Ghosh, and K. Grauman, “Discovering important people
and objects for egocentric video summarization,” in Proceedings of the
IEEE Conference on Computer Vision and Pattern Recognition, 2012,
pp. 1346–1353.
[4] G. Kim, L. Sigal, and E. P. Xing, “Joint summarization of large-scale
collections of web images and videos for storyline reconstruction,” in
Proceedings of the IEEE Conference on Computer Vision and Pattern
Recognition, 2014, pp. 4225–4232.
[5] B. Zhao and E. P. Xing, “Quasi real-time summarization for consumer
videos,” in Proceedings of the IEEE Conference on Computer Vision
and Pattern Recognition, 2014, pp. 2513–2520.
[6] A. Khosla, R. Hamid, C.-J. Lin, and N. Sundaresan, “Large-scale video
summarization using web-image priors,” in Proceedings of the IEEE
Conference on Computer Vision and Pattern Recognition, 2013, pp.
2698–2705.
[7] P. Mundur, Y. Rao, and Y. Yesha, “Keyframe-based video summarization
using delaunay clustering,” International Journal on Digital Libraries,
vol. 6, no. 2, pp. 219–232, 2006.
[8] A. A. Tonge and S. D. Thepade, “Key frame extraction for video content summarization using orthogonal transforms and fractional energy
coefficients,” in Proceedings of the IEEE conference on Information
Processing, 2015, pp. 642–646.
[9] R. Laganière, R. Bacco, A. Hocevar, P. Lambert, G. Paı̈s, and B. E.
Ionescu, “Video summarization from spatio-temporal features,” in Proceedings of the 2nd ACM TRECVid Video Summarization Workshop,
2008, pp. 144–148.
[10] J. Nam and A. H. Tewfik, “Event-driven video abstraction and visualization,” Multimedia Tools and Applications, vol. 16, no. 1, pp. 55–77,
2002.
[11] Z. Lu and K. Grauman, “Story-driven summarization for egocentric
video,” in Proceedings of the IEEE Conference on Computer Vision
and Pattern Recognition, 2013, pp. 2714–2721.
[12] Y.-L. Lin, V. I. Morariu, and W. Hsu, “Summarizing while recording:
Context-based highlight detection for egocentric videos,” in Proceedings
of the IEEE International Conference on Computer Vision Workshops,
2015, pp. 51–59.
[13] J. Meng, H. Wang, J. Yuan, and Y.-P. Tan, “From keyframes to
key objects: Video summarization by representative object proposal
selection,” in Proceedings of the IEEE Conference on Computer Vision
and Pattern Recognition, 2016, pp. 1039–1048.
[14] B. Gong, W.-L. Chao, K. Grauman, and F. Sha, “Diverse sequential
subset selection for supervised video summarization,” in Advances in
Neural Information Processing Systems, 2014, pp. 2069–2077.
[15] M. Gygli, H. Grabner, and L. Van Gool, “Video summarization by
learning submodular mixtures of objectives,” in Proceedings of the IEEE
Conference on Computer Vision and Pattern Recognition, 2015, pp.
3090–3098.
[16] K. Zhang, W.-L. Chao, F. Sha, and K. Grauman, “Summary transfer:
Exemplar-based subset selection for video summarization,” in Proceedings of the IEEE Conference on Computer Vision and Pattern
Recognition, 2016, pp. 1059–1067.
[17] D. Liu, G. Hua, and T. Chen, “A hierarchical visual model for video
object summarization,” IEEE transactions on pattern analysis and
machine intelligence, vol. 32, no. 12, pp. 2178–2190, 2010.
[18] Y.-F. Ma, L. Lu, H.-J. Zhang, and M. Li, “A user attention model for
video summarization,” in Proceedings of the tenth ACM international
conference on Multimedia, 2002, pp. 533–542.
[19] C.-W. Ngo, Y.-F. Ma, and H.-J. Zhang, “Automatic video summarization
by graph modeling,” in Proceedings of the IEEE international conference on computer vision, 2003, pp. 104–109.

13

[20] T. Yao, T. Mei, and Y. Rui, “Highlight detection with pairwise deep
ranking for first-person video summarization,” in Proceedings of the
IEEE Conference on Computer Vision and Pattern Recognition, 2016,
pp. 982–990.
[21] Y. Song, J. Vallmitjana, A. Stent, and A. Jaimes, “Tvsum: Summarizing
web videos using titles,” in Proceedings of the IEEE Conference on
Computer Vision and Pattern Recognition, 2015, pp. 5179–5187.
[22] S. Zhang, Y. Zhu, and A. K. Roy-Chowdhury, “Context-aware surveillance video summarization,” IEEE Transactions on Image Processing,
vol. 25, no. 11, pp. 5469–5478, 2016.
[23] F. Zhou, S. Bing Kang, and M. F. Cohen, “Time-mapping using spacetime saliency,” in Proceedings of the IEEE Conference on Computer
Vision and Pattern Recognition, 2014, pp. 3358–3365.
[24] M. Sun, A. Farhadi, B. Taskar, and S. Seitz, “Salient montages from
unconstrained videos,” in European Conference on Computer Vision,
2014, pp. 472–488.
[25] Y. Pritch, A. Rav-Acha, A. Gutman, and S. Peleg, “Webcam synopsis:
Peeking around the world,” in Proceedings of the IEEE international
conference on computer vision, 2007, pp. 1–8.
[26] D. Zhang, J. Han, L. Jiang, S. Ye, and X. Chang, “Revealing event
saliency in unconstrained video collection,” IEEE Transactions on Image
Processing, vol. 26, no. 4, pp. 1746–1758, 2017.
[27] W.-S. Chu, Y. Song, and A. Jaimes, “Video co-summarization: Video
summarization by visual co-occurrence,” in Proceedings of the IEEE
Conference on Computer Vision and Pattern Recognition, 2015, pp.
3584–3592.
[28] S. Hochreiter and J. Schmidhuber, “Long short-term memory,” Neural
computation, vol. 9, no. 8, pp. 1735–1780, 1997.
[29] X. Liang, X. Shen, D. Xiang, J. Feng, L. Lin, and S. Yan, “Semantic object parsing with local-global long short-term memory,” in Proceedings
of the IEEE Conference on Computer Vision and Pattern Recognition,
2016, pp. 3185–3193.
[30] X. Liang, X. Shen, J. Feng, L. Lin, and S. Yan, “Semantic object parsing
with graph lstm,” in European Conference on Computer Vision, 2016,
pp. 125–143.
[31] X. Liang, L. Lin, X. Shen, J. Feng, S. Yan, and E. P. Xing, “Interpretable
structure-evolving lstm,” arXiv preprint arXiv:1703.03055, 2017.
[32] N. Srivastava, E. Mansimov, and R. Salakhudinov, “Unsupervised learning of video representations using lstms,” in International Conference
on Machine Learning, 2015, pp. 843–852.
[33] P. Pan, Z. Xu, Y. Yang, F. Wu, and Y. Zhuang, “Hierarchical recurrent
neural encoder for video representation with application to captioning,”
in Proceedings of the IEEE Conference on Computer Vision and Pattern
Recognition, 2016, pp. 1029–1038.
[34] G. Li, S. Ma, and Y. Han, “Summarization-based video caption via
deep neural networks,” in Proceedings of the 23rd ACM international
conference on Multimedia, 2015, pp. 1191–1194.
[35] H. Yang, B. Wang, S. Lin, D. Wipf, M. Guo, and B. Guo, “Unsupervised
extraction of video highlights via robust recurrent auto-encoders,” in
Proceedings of the IEEE international conference on computer vision,
2015, pp. 4633–4641.
[36] A. Graves and N. Jaitly, “Towards end-to-end speech recognition with
recurrent neural networks.” in International Conference on Machine
Learning, vol. 14, 2014, pp. 1764–1772.
[37] K. Zhang, W.-L. Chao, F. Sha, and K. Grauman, “Video summarization
with long short-term memory,” in European conference on computer
vision, 2016, pp. 766–782.
[38] B. A. Olshausen and D. J. Field, “Sparse coding of sensory inputs,”
Current opinion in neurobiology, vol. 14, no. 4, pp. 481–487, 2004.
[39] A. Ng, “Sparse autoencoder,” CS294A Lecture notes, vol. 72, no. 2011,
pp. 1–19, 2011.
[40] J. Han, D. Zhang, X. Hu, L. Guo, J. Ren, and F. Wu, “Background priorbased salient object detection via deep reconstruction residual,” IEEE
Transactions on Circuits and Systems for Video Technology, vol. 25,
no. 8, pp. 1309–1321, 2015.
[41] X. Liang, S. Liu, Y. Wei, L. Liu, L. Lin, and S. Yan, “Towards
computational baby learning: A weakly-supervised approach for object
detection,” in Proceedings of the IEEE international conference on
computer vision, 2015, pp. 999–1007.
[42] X. Wang and A. Gupta, “Unsupervised learning of visual representations
using videos,” in Proceedings of the IEEE international conference on
computer vision, 2015, pp. 2794–2802.
[43] X. Liang, Y. Wei, L. Lin, Y. Chen, X. Shen, J. Yang, and S. Yan,
“Learning to segment human by watching youtube,” IEEE transactions
on pattern analysis and machine intelligence, vol. 39, no. 7, pp. 1462–
1468, 2017.

[44] Y. Xiang, A. Alahi, and S. Savarese, “Learning to track: Online multiobject tracking by decision making,” in Proceedings of the IEEE
international conference on computer vision, 2015, pp. 4705–4713.
[45] R. Girshick, “Fast r-cnn,” in Proceedings of the IEEE international
conference on computer vision, 2015, pp. 1440–1448.
[46] M. Everingham, L. Van Gool, C. K. I. Williams, J. Winn,
and A. Zisserman, “The PASCAL Visual Object Classes
Challenge
2012
(VOC2012)
Results,”
http://www.pascalnetwork.org/challenges/VOC/voc2012/workshop/index.html.
[47] M. A. Smith and T. Kanade, “Video skimming and characterization
through the combination of image and language understanding,” in
Proceedings of the 1998 International Workshop on Content-Based
Access of Image and Video Databases, 1998, pp. 61–70.
[48] D. Li, J.-B. Huang, Y. Li, S. Wang, and M.-H. Yang, “Weakly supervised
object localization with progressive domain adaptation,” in Proceedings
of the IEEE Conference on Computer Vision and Pattern Recognition,
2016, pp. 3512–3520.
[49] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, “Imagenet:
A large-scale hierarchical image database,” in Proceedings of the IEEE
Conference on Computer Vision and Pattern Recognition, 2009, pp. 248–
255.
[50] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image
recognition,” in Proceedings of the IEEE Conference on Computer Vision
and Pattern Recognition, 2016, pp. 770–778.
[51] O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma,
Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, A. C. Berg, and L. FeiFei, “Imagenet large scale visual recognition challenge,” International
Journal of Computer Vision, vol. 115, no. 3, pp. 211–252, 2015.
[52] Y. Li and B. Merialdo, “Multi-video summarization based on videommr,” in Proceedings of the IEEE International Workshop on Image
Analysis for Multimedia Interactive Services, 2010, pp. 1–4.
[53] S. E. F. De Avila, A. P. B. Lopes, A. da Luz, and A. de Albuquerque Araújo, “Vsumm: A mechanism designed to produce static
video summaries and a novel evaluation method,” Pattern Recognition
Letters, vol. 32, no. 1, pp. 56–68, 2011.
[54] K. Cho, B. Van Merriënboer, C. Gulcehre, D. Bahdanau, F. Bougares,
H. Schwenk, and Y. Bengio, “Learning phrase representations using
rnn encoder-decoder for statistical machine translation,” arXiv preprint
arXiv:1406.1078, 2014.
[55] D. Zhang, D. Meng, and J. Han, “Co-saliency detection via a self-paced
multiple-instance learning framework,” IEEE transactions on pattern
analysis and machine intelligence, vol. 39, no. 5, pp. 865–878, 2017.
[56] D. Zhang, J. Han, C. Li, J. Wang, and X. Li, “Detection of co-salient
objects by looking deep and wide,” International Journal of Computer
Vision, vol. 120, no. 2, pp. 215–232, 2016.
[57] J. Zhang, Z. Lin, J. Brandt, X. Shen, and S. Sclaroff, “Top-down neural
attention by excitation backprop,” in European Conference on Computer
Vision. Springer, 2016, pp. 543–559.
[58] Y. Yuan, X. Liang, X. Wang, D.-Y. Yeung, and A. Gupta, “Temporal
dynamic graph lstm for action-driven video object detection,” arXiv
preprint arXiv:1708.00666, 2017.
[59] X. Liang, L. Lee, W. Dai, and E. P. Xing, “Dual motion gan for futureflow embedded video prediction,” arXiv preprint arXiv:1708.00284,
2017.

