1

Summarization of User-Generated Sports Video
by Using Deep Action Recognition Features

arXiv:1709.08421v1 [cs.CV] 25 Sep 2017

Antonio Tejero-de-Pablos, Member, IEEE, Yuta Nakashima, Member, IEEE, Tomokazu Sato, Member, IEEE,
Naokazu Yokoya, Life Senior Member, IEEE, Marko Linna, and Esa Rahtu

Abstract—Automatically generating a summary of sports video
poses the challenge of detecting interesting moments, or highlights, of a game. Traditional sports video summarization methods leverage editing conventions of broadcast sports video that
facilitate the extraction of high-level semantics. However, usergenerated videos are not edited, and thus traditional methods
are not suitable to generate a summary. In order to solve this
problem, this work proposes a novel video summarization method
that uses players’ actions as a cue to determine the highlights
of the original video. A deep neural network-based approach
is used to extract two types of action-related features and to
classify video segments into interesting or uninteresting parts.
The proposed method can be applied to any sports in which
games consist of a succession of actions. Especially, this work
considers the case of Kendo (Japanese fencing) as an example of
a sport to evaluate the proposed method. The method is trained
using Kendo videos with ground truth labels that indicate the
video highlights. The labels are provided by annotators possessing
different experience with respect to Kendo to demonstrate how
the proposed method adapts to different needs. The performance
of the proposed method is compared with several combinations
of different features, and the results show that it outperforms
previous summarization methods.

Video summary

Index Terms—Sports video summarization; user-generated
video; action recognition; deep learning; 3D convolutional neural
networks; long short-term memory.

I. I NTRODUCTION

T

HE widespread availability of cameras has led to an
enormous and ever-growing collection of unedited and
unstructured videos generated by users around the world
[1]. A popular domain corresponds to sports videos taken at
public events and professional/amateur matches. These types
of user-generated sports videos (UGSVs) are often lengthy
with several uninteresting parts, and thus many of them are
stored and are never reviewed. A convenient way to review,
transfer, and share the video via channels, such as social
network services, includes generating summaries of a UGSV
that only shows the interesting parts or highlights.
Automatic video summarization is a challenging problem
that involves extracting semantics from video. Traditional usergenerated video summarization methods target general videos
in which contents are not limited to a specific domain because
of the difficulty in extracting semantics from an unstructured
A. Tejero-de-Pablos, Y. Nakashima, T. Sato and N. Yokoya were with
the Graduate School of Information Science, Nara Institute of Science and
Technology, Nara, 630-0101 Japan. e-mail: antonio.tejero.ao4@is.naist.jp.
M. Linna is with the Center for Machine Vision Research, University
of Oulu, Oulu, 90014 Finland. E. Rahtu is with the Department of Signal
Processing, Tampere University of Technology, Tampere, 33101 Finland
Manuscript received July, 2017

Fig. 1. An overview of the proposed method to generate a summary of
user-generated sports video (UGSV) based on players’ actions. Two types of
features that represent players’ actions, namely body joint-based and holistic
actions, are used to extract highlights from the original video.

video. As opposed to extracting semantics, these methods
use low-level visual features and attempt to reduce visual redundancy using clustering-based approaches [2]. More recent
user-generated video summarization methods use deep neural
network-based features to extract higher-level semantics [3].
With respect to sports video and especially with respect
to professional sports in broadcast TV programs, there exist
a number of summarization methods that leverage editing
conventions to extract high-level semantics by exploiting a
knowledge of the specific sport [4], [5]. For example, broadcast sports video contains slow-motion replays [6], narration
and superimposed text [7], and specific camera work [8].
This type of video editing constitutes the basis for heuristic
rules that aid in the determination of highlights (or certain
interesting moments of a game such as a free kick in soccer
or a pitch in baseball). Additionally, broadcast video is often
edited by following the structure of the sport (i.e., “downs”

2

in American football), and this constitutes another cue for
summarization [9].
UGSV lies in in-between general user-generated video and
broadcast sports video. Given a specific sport, domain knowledge can be used to generate a UGSV summary. However,
UGSV does not typically follow any editing convention or
structure, and thus a different type of cues is required to grab
the semantics.
This paper describes a novel method for UGSV summarization. Our observation with respect to semantics extraction is
that a game in most sports consists of a succession of players’
actions, and thus the actions can be one of the most important
cues to determine if a certain part of video is interesting or
not. For example, a definitive smash in tennis is more likely to
be enjoyed by tennis viewers than a repetitive ball exchange.
Also, a feint in boxing might not be interesting by itself, but
viewers would surely enjoy it if it is followed by an uppercut
that knocks out the opponent. Based on this observation, the
proposed method uses players’ actions to model the highlights
of a sports game (Fig. 1).
Inspired by recent methods for action recognition in video,
the proposed method uses a two-stream architecture that
extracts two types of action features for action representation.
One type involves players’ body joint positions estimated in
2D or 3D (obtainable from depth maps). Body joint-based
features provide a precise representation of actions. The other
type involves holistic features that can be obtained with deep
convolutional neural networks (CNNs) designed to extract
spatio-temporal features from video. Holistic features help to
capture actions in their context. Subsequently, long short-term
memory (LSTM) is used to model the temporal dependencies
of the extracted features for highlight classification. In our
summaries, a highlight may contain one or more actions
performed by the players. Several types of body joint-based
and holistic features are comparatively evaluated for UGSV
summarization.
We consider the case of Kendo (Japanese fencing) as an
example of a sport to evaluate the proposed method. This work
is an extension of our previous work in [10].
The main contributions of this work are as follows:
•

•

•

•

A novel UGSV summarization method that determines
highlights in a video by using players’ action features
and a deep neural network.
A comparison of several action feature extraction methods, i.e., body joint features (RGB image-based 2D
joint positions and depth map-based 3D joint positions)
and holistic features (C3D [11] and CNN-ISA [12]) to
demonstrate their adequacy to model video highlights.
A new UGSV dataset with 246 min of Kendo videos in
which each second of a video has a label that indicates
whether or not it is a part of a highlight. The labels were
provided by annotators with and without experience in
Kendo.
Objective and subjective evaluations of the proposed
method. Users with and without experience in Kendo
were surveyed to investigate the adequacy of the proposed
method with respect to individual needs.

II. R ELATED WORK
This section introduces existing video summarization methods in terms of the types of video (i.e., broadcast sports video
and user-generated video). This section also reviews existing
work in action recognition, which constitutes a key technique
for modeling highlights in the proposed method.
A. Action recognition from video
Traditional approaches for action recognition use conventional classifiers and hand-crafted features that represent human movement in RGB video [13], [14], [15]. However,
the field of action recognition recently benefits from deep
neural networks for feature extraction and classification. An
example corresponds to three-dimensional convolutional neural networks (3D CNNs) that constitute an extension of CNNs
applied to images (2D CNNs). While 2D CNNs perform only
spatial operations in a single image, 3D CNNs also perform
temporal operations while preserving temporal dependencies
among the input video frames [11]. Le et al. [12] used a 3D
CNN with independent subspace analysis (CNN-ISA) and a
support vector machine (SVM) to recognize human actions
from video. Additionally, Tran et al. [11] designed a CNN
called C3D to extract video features that were subsequently
fed to an SVM for action recognition.
Another state-of-the-art CNN-based action recognition
method employed two types of streams, namely a spatial
appearance stream and a temporal motion stream [16], [17].
Videos are decomposed into spatial and temporal components,
i.e., into an RGB and optical flow representation of its frames,
and fed into two separate 3D CNNs. Each stream separately
provides a score for each possible action, and the scores from
two streams were later combined to obtain a final decision.
This architecture is supported by the two-stream hypothesis of
neuroscience in which the human visual system is composed of
two different streams in the brain, namely the dorsal stream
(spatial awareness and guidance of actions) and the ventral
stream (object recognition and form representation) [18].
In addition to RGB videos, other methods leverage depth
maps obtained from commodity depth sensors (e.g. Microsoft
Kinect) to estimate the human 3D pose for action recognition
[19], [20], [21]. The third dimension provides robustness to
occlusions and variations from the camera viewpoint.
B. Broadcast sports video summarization
Summarization of sports video focuses on extracting interesting moments (i.e., highlights) of a game. A major approach
leverages editing conventions such as those present in broadcast TV programs. Editing conventions are common to almost
all videos of a specific sport and allow automatic methods
to extract high-level semantics [22], [23]. Ekin et al. [7]
summarized broadcast soccer games by leveraging predefined
camera angles in edited video to detect soccer field elements
(e.g., goal posts). Similar work used slow-motion replays to
determine key events in a game [6] and predefined camera
motion patterns to find scenes in which players scored in
basketball/soccer games [8].

3

In addition to editing conventions, the structure of the sport
also provides high-level semantics for summarization. Certain
sports are structured in “plays” that are defined based on
the rules of the sport and are often easily recognized in
broadcast videos [4], [24], [25]. For example, Li et al. [9]
summarized American football games by leveraging their turnbased structure and recognizing “down” scenes from the video.
Other methods used metadata in sports videos [5], [26] since it
contains high-level descriptions (e.g., “hits” may be annotated
in the metadata with their timestamps for a baseball game). A
downside of these methods is that they cannot be applied to
sports video without any editing conventions, structures, and
metadata. Furthermore, they are based on heuristics, and thus
it is difficult to generalize them to different sports.
Existing work also proposed several methods that are not
based on heuristics. These methods leverage variations between scenes that are found in broadcast video (e.g., the closeup in a goal celebration in soccer). Chen et al. [27] detected
intensity variations in color frames to segment relevant events
to summarize broadcast videos of soccer, basketball, and tennis. Mendi et al. [28] detected the extrema in the optical flow
of a video to extract the frames with the highest action content
and construct a summary for broadcast rugby video. These
methods can be more generally applied to broadcast videos,
but they lack high-level semantics, and thus the extracted
scenes do not always correspond to the highlights of the game.
C. User-generated video summarization
Sports video includes a somewhat universal criterion on the
extent to which a “play” is interesting (e.g., a homerun in a
baseball game should be an interesting play for most viewers).
In contrast, user-generated video in general do not have a clear
and universal criterion to identify interesting moments. Additionally, neither editing conventions nor specific structures that
can be used to grab high-level semantics can be leveraged [29].
Hence, many video summarization methods for user-generated
video are designed to reduce the redundancy of a lengthy
original video as opposed to determining interesting moments.
Traditional methods uniformly sample frames [30] or cluster
them based on low-level features, such as color [2], to extract a
brief synopsis of a lengthy video. These methods do not extract
highlights of the video, and therefore researchers proposed
other types of summarization criteria such as important objects
[31], attention [32], [33], and interestingness [34].
Recent methods use deep neural networks to automatically
learn a criterion to model highlights. Yang et al. [35] extracted
features from ground-truth video summaries to train a model
for highlight detection. Otani et al. [36] use a set of both
original videos and their textual summaries that are generated
via majority voting by multiple annotators to train a model to
find video highlights. Video titles [37] and descriptions [3] can
also be used to learn a criterion to generate summaries. The
aforementioned methods employed networks with CNNs and
LSTMs, and this requires a large amount of data for training.
The generation of these types of large summarization datasets
for training their network is non-viable for most researchers,
and thus their models are built on pre-trained networks such
as VGG [38] and GoogLeNet [39].

III. UGSV SUMMARIZATION USING ACTION FEATURES
UGSV summarization inherits the intricacies of usergenerated video summarization. The extraction of high-level
semantics is not trivial in the absence of editing conventions.
However, given a specific sport, it is possible to leverage
domain knowledge to facilitate the extraction of high-level
semantics. The idea in the present work for semantics extraction involves utilizing players’ actions, as they are the
main constituents of a game. Our previous work [10] applied
an action recognition technique to sports videos to determine
combinations of actions that interest viewers by using a hidden
Markov model with Gaussian Mixture emissions. To the best
of our knowledge, this work was the first to use a UGSV
summarization based on players’ actions.
A major drawback of the previous work [10] involves the
usage of the outputs of a classic action recognizer as features
to determine the highlights of the UGSV. Moreover, in addition
to the UGSV summarization dataset, the method also requires
an action dataset of the sport to train the action recognizer.
Another drawback of [10] is that it only uses features from
3D joint positions (that are estimated by, e.g., [40]). They
provide rich information on players’ actions but miss other
potential cues for summarization contained in the appearance
of the scene. For example, appearance is useful when the joint
position estimation fails.
In this work, the method in [10] is extended by employing
a two-stream deep neural network [16], [17]. Our new method
considers two different types of inputs, namely RGB frames
of video and body joint positions, and each of them are transformed through two separate neural networks (i.e., streams).
These two streams are then fused to form a single action
representation to determine the highlights. Our method does
not require an action recognizer; and the proposed network
is trained from the lower layer to the top layers by using a
UGSV summarization dataset.
Given the proposed method, it is necessary for the target
sports to satisfy the following conditions: (1) a game consists
of a series of recognizable actions performed by each player
and (2) players are recorded from a close distance for joint
position estimation. However, it is expected that the idea of
using action recognition-related features for UGSV summarization is still valid for most types of sports.
A. Overview
In this work, UGSV summarization is formulated as a
problem of classifying a video segment in the original video
as interesting (and thus included in the summary) or uninteresting. A two-stream neural network is designed for this
problem, and it is trained in a supervised manner with ground
truth labels provided by multiple annotators.
Figure 1 shows an overview of the proposed method.
The method first divides the original input video into a set
S = {st } of video segments in which RGB frames can be
accompanied by their corresponding depth maps. A video
segment st is then fed into the two-stream network. The body
joint-based feature stream considers RGB frames (and depth
maps) in st to obtain body joint-based features xt , and the

4

...
Time
Fig. 2. In the video segmentation, a video segment st contains frames inbetween t − 1 and t + 2 sec. Each video segment overlaps with adjacent ones
for 2 sec.

holistic feature stream computes holistic features yt from the
RGB frames. The former stream captures the players’ motion
in detail by explicitly estimating their body joint positions. The
latter stream represents entire frames in the video segment, and
this is helpful to encode, for example, the relationship between
the players. The features X = {xt } and Y = {yt } are then
used for highlight classification by considering the temporal
dependencies among the video segments. The highlight summaries correspond to a concatenation of the segments that are
classified as interesting.
B. Video segmentation
Various methods have been proposed to segment a video
(e.g., its content [27]). In the proposed method, the original
input video of length T seconds (sec) is uniformly segmented
into multiple overlapping segments, so that subsequently a
second t of video can be represented by extracting action
features from a video segment st , i.e., S = {st |t = 1, . . . , T }.
Thus, T also corresponds to the number of video segments
in S, and st corresponds to the video segment that contains
frames from sec t − 1 to sec t + τ − 1. For a finer labeling
of highlights, short video segments are required. We choose
a τ = 3, for which adjacent video segments overlap by 2
sec as shown in Fig. 2. Each segment st may contain a
different number of frames, especially when the input video is
captured with an RGB-D camera (e.g., Microsoft Kinect) due
to automatic exposure control.
C. Body joint-based feature stream
In this stream (Fig. 3), a sequence of positions of the
players’ body joints (e.g., head, elbow, etc.) that represent the
movement of the players irrespective of their appearance is
used to obtain a detailed representation of players’ actions.
Specifically, two types of joint representations are employed
in this work, namely 3D positions from depth maps or 2D
positions from RGB frames.
With respect to the 3D body joint positions, the skeleton
tracker (e.g., [40]) is used as in the previous work [10], and it
estimates 3D positions from depth maps. The 3D positions are
usually represented in the camera coordinate system, and thus
they are view-dependent, thereby introducing extra variations.

Fig. 3. In the body joint-based feature stream, an LSTM is fed with the body
joint positions estimated from players on each frame uft to model temporal
dependencies and extract a feature vector ht . Additionally, these body joint
positions are also used to calculate an activity measure for all players at . The
body joint-based feature vector is their concatenation xt .

Therefore, the 3D positions from the camera coordinate system
are transformed to each player’s coordinate system in which
the origin corresponds to one of the body joints (e.g., torso).
In the absence of depth maps (which is likely in current
user-generated video), 2D body joint positions can still be
estimated from RGB frames. Recent methods in human pose
estimation leverage 2D CNNs to learn spatial relationships
among human body parts and estimate 2D joint positions
[41]. These types of 2D positions are not as robust relative
to view variations as 3D positions. However, they can be
extracted from RGB frames alone without using depth maps.
The given 2D body joint positions are also transformed to
positions relative to the player’s coordinate system to ensure
that they are translation invariant.
The use of an activity measure works positively while
extracting highlights [10]. In order to calculate the activity
measure of a certain player q in the video segment s (we
omit subscript t in this subsection for notation simplicity), the
volume (or plane for the 2D case) around the player is divided
into a certain number of regions, and the ratio rv of the number
of frames in the video segment in which the joint j falls into
region v is calculated. The activity measure aq is defined as
the entropy obtained based on rv . With respect to each joint
j in player q’s body, we compute the entropy as follows:
X
ej = −
rv log(rv ).
(1)
v

Then, we calculate the activity measure for player q as follows:
aq =

J
X
j=1

ej .

(2)

5

The activity measure for all players in a segment is calculated.
More details on the activity measure can be found in [10].
Let ufqj in R3 or R2 (a row vector) denote the 3D or 2D
relative position of joint j of player q in frame f of video
segment s. Subsequently, given the number of players Q and
of estimated body joints J, the concatenation of the body joints
of all players in frame f is defined as follows:
uf = (uf11 · · · ufqj · · · ufQJ ).

(3)

As shown in Fig. 3, vectors u1 to uF are passed through
an LSTM to model the temporal dependencies of the joint
positions of players’ bodies in s. After feeding the last vector
uF , the hidden state vector h of the LSTM is considered as a
representation of {uf }. The state of the LSTM is reset to all
zeros prior to feeding the next video segment. It is assumed
that the number of players Q does not change. However, some
players can be out of the field-of-view of the camera. In this
case, the corresponding elements in u and a are substituted
with zeros.
The proposed method represents a video segment s by
concatenating the LSTM output and the activity measure of
all players in one vector as follows:

Fig. 4. The recurrent neural network architecture for highlight classification
consists of a single LSTM layer and several fully-connected layers. The body
joint-based features xt and holistic features yt extracted from video segment
st are input to calculate the probability pt that the segment is interesting.

where a denotes the concatenation of (a1 · · · aQ ).

This stream represents a video segment s by using a holistic
feature vector y that corresponds to the output of one of the
aforementioned 3D CNNs.

D. Holistic feature stream

E. Highlight classification using LSTM

This stream encodes a video segment s in a spatio-temporal
representation. We rely on state-of-the-art 3D CNNs over RGB
frames. Training a 3D CNN from scratch requires thousands of
videos [42] that are not available for the proposed task. Recent
work on deep neural networks for computer vision [11], [17],
[43] shows that the activations of an upper layer of a CNN
are useful for other related tasks without requiring fine-tuning.
Thus, 3D CNN in which parameters are pre-trained with largescale datasets can be used instead to leverage a huge amount
of labeled training data [44]. The proposed method utilizes
a 3D CNN for action feature extraction pre-trained with a
publicly available action recognition dataset, such as Sports1M [42]. Unlike our previous work [10], which required to
classify players’ actions, it is not necessary to use a sportspecific action recognition dataset.
Two types of holistic representations of video segments
extracted using 3D CNNs are employed, namely CNN-ISA
[12] and C3D [11]. Specifically, CNN-ISA provides a representation robust to local translation (e.g., small variations in
players’ or camera motion) while it is selective to frequency,
rotation, and velocity of such motion. The details of CNNISA can be found in [12]. CNN-ISA achieved state-of-theart performance in well-known datasets for action recognition
such as YouTube [45], Hollywood2 [46], and UCF sports [47].
Additionally, C3D features provide a representation of objects,
scenes, and actions in a video. The network architecture and
other details can be found in [11]. C3D pre-trained with the
Sports-1M dataset achieved state-of-the-art performance on
action recognition over the UCF101 dataset [48].

Figure 4 shows the network architecture designed to extract
highlights of UGSV using the features xt and yt from video
segment st . The temporal dependencies among video segments
are modeled using an LSTM, and the network outputs the
probability pt that the video segment st is interesting. First,
the features are concatenated to form vector zt = (xt yt ).
Vector zt then goes through a fully-connected layer to reduce
its dimensionality.
It is assumed that interesting video segments are related to
each other in time, in the same way a skillful boxer first feints
a punch prior to hitting to generate an opening in the defense.
Existing work in video summarization uses LSTMs to extract
video highlights [35] since it allows the modeling of temporal
dependencies across longer time periods when compared to
other methods [49]. Following this concept, an LSTM layer
is introduced to the network for highlight classification. The
hidden state of the LSTM from each time step goes through
two fully-connected layers, and this results in a final softmax
activation of two units corresponding to “interesting” and
“uninteresting.”
The proposed method provides control over the length L
of the output summary. The softmax activation of the unit
corresponding to “interesting” is considered as the probability
pt ∈ [0, 1] that segment st is interesting, and skimming curve
formulation [50] is applied to the sequence of probabilities by
decreasing a threshold θ from 1 until a set of segments whose
total length is highest below L is determined (Fig. 5). The
segments in which the probability exceeds θ are concatenated
to generate the output summary in the temporal order. Hence,

x = (h a),

(4)

6

Fig. 5. A summary is generated by concatenating segments in which the
probability pt of being a highlight surpasses a certain threshold θ. The
threshold is chosen to fit the summary length.

the resulting summary may contain multiple consecutive interesting segments.
F. Network training
A pre-trained CNN is used in the holistic features stream
(i.e., CNN-ISA or C3D) while the LSTMs and fully-connected
layers are trained from scratch. Hence, during training, the
parameters in the holistic feature stream (i.e., CNN layers)
are fixed and those in the body joint-based feature stream (i.e.,
lstmJ ) and highlight classification (i.e., fc1, lstmH , fc2, and
fc3) are updated.
The UGSV dataset contains video and ground truth labels
lt ∈ {0, 1} for every second, where lt = 1 implies that the
period from t sec to t + 1 sec of the video is “interesting”
and lt = 0 otherwise. Label lt is assigned to its corresponding
video segment st .
With respect to training, cross-entropy loss ` is used:
X
`=
lt log pt .
(5)
IV. E XPERIMENTS
The proposed method is evaluated objectively and subjectively. With respect to the objective evaluation, the performance of the proposed method is compared while using
different representations of the players’ actions. Specifically,
only body joint features (3D or 2D), only holistic motion
features (CNN-ISA or C3D), and a combination of both
features are evaluated. Subsequently, the completeness of the
highlights of the generated summaries are examined. With
respect to the subjective evaluation, users with and without
experience in the sport are surveyed to study their opinions
with respect to the summaries.
A. Implementation details
For the evaluation, Kendo (Japanese fencing) was selected
as an example of a sport. Kendo is a martial art featuring two
players and a set of recognizable actions (e.g., attacking and
parrying). We used the UGSV Kendo dataset in [10], which
contains 90 min of self-recorded Kendo matches divided in 10
RGB-D videos taken with a Microsoft Kinect v2, and extended
it by adding 18 more self-recorded RGB-D Kendo videos.
The total length of the videos is 246 min with a framerate of
approximately 20 fps.

The body joint-based feature stream was configured for Q =
2 players since Kendo is a two-player sport. The tracker in [40]
was used to estimate J = 15 3D body joint positions from
depth maps: head, neck, torso, right shoulder, right elbow,
right wrist, left shoulder, left elbow, left wrist, right hip, right
knee, right ankle, left hip, left knee, and left ankle. In order to
estimate the 2D positions of the players’ joints from the RGB
frames, the CNN-based method proposed by Linna et al. [51]
was used. We pre-trained this joint estimation CNN with the
human pose dataset used by Linna et al. [51], and then we
fine-tuned it with our extended UGSV Kendo video dataset.
The network provides J = 13 joints (that is the same as the
3D case with the exception of neck and torso). Therefore, the
size of vector uft is Q×J ×3 = 90 in the case of 3D positions
and Q × J × 2 = 52 in the case of 2D. Given that the size
of lstmJ is the same as that of the input and that the size of
at is Q = 2, the feature vector xt for the stream is ∈ R92 for
3D and ∈ R54 for 2D.
With respect to the holistic feature stream, either the CNNISA [12] or C3D [11] networks were used. The UGSV
Kendo dataset is not sufficiently large to train the CNNs
from scratch, and thus networks pre-trained with an action
recognition dataset were used. The CNN-ISA was trained
in an unsupervised way with the Hollywood2 dataset that
consists of 2859 videos [46]. For this network, we followed
the configuration in [52]. We used a vector quantization
representation of the extracted features with a codebook size
of 400, thereby resulting in a feature vector yt ∈ R400 for
each segment st . The C3D was trained with the Sports-1M
dataset [42] that consisted of 1.1 million videos of sports
activities. The C3D features were extracted as indicated in [11]
by uniformly sub-sampling 16 frames out of approximately
60 frames in st (the number of frames in st may vary for
different segments due to the variable framerate of Microsoft
Kinect v2) and subsequently the activations from layer fc6
(i.e., yt ∈ R4096 ) were extracted.
The proposed method was implemented in Chainer [53].
The learning rate was calculated by the adaptive moment
estimation algorithm (Adam) [54] with α = 0.001. Sigmoid
activation was introduced after the fully-connected layers.
Table I summarizes the number of learnable parameters for
each layer, which varies based on the choice of features.
B. Results
For annotating the ground truth, 15 participants were invited
and divided into two groups, namely experienced (E, 5 people)
and inexperienced (NE, 10 people), based on their experience
in the target sport (i.e., Kendo). It was assumed that the
highlights preferred by the E and NE groups would exhibit significant variations, and an aim of the study included evaluating
the extent to which the proposed method adapts to the needs
of each group. For this, the participants annotated manually
the highlights of the 28 videos. The ground truth labels of the
videos were separately obtained for both E and NE groups.
With respect to each one-second period t of video, the ground
truth label is lt = 1 if at least 40% of the participants annotated
it as interesting (i.e., 2 people in group E and 4 people in

7

TABLE I
S IZE OF THE LEARNABLE ELEMENTS IN THE NETWORK WITH RESPECT TO THE FEATURES USED (input × output).
F EATURE VECTOR SIZES ARE DETAILED IN S ECTION IV-A)

lstmJ
fc1
lstmH
fc2
fc3

Body
3D joints
90 × 90
92 × 50
50 × 50
50 × 20
20 × 2

joint-based features only
2D joints
Action recognition
52 × 52
—
54 × 50
402 × 400
50 × 50
400 × 400
50 × 20
400 × 100
20×2
100 × 2

Holistic features only
CNN-ISA
C3D
—
—
400 × 400
4096 × 400
400 × 400
400 × 400
400 × 100
400 × 100
100 × 2
100 × 2

group NE). Otherwise, lt = 0. Due to group E’s technical
knowledge of Kendo, their highlights contain very specific
actions (e.g., decisive strikes and counterattacks). Conversely,
group NE selected strikes as well as more general actions (e.g.,
parries and feints), and thus their labeled highlights are almost
three times as long as group E’s (please refer to the durations
in Appendix A).
The network was separately trained with each group’s
ground truth labels in the leave-one-out (LOO) fashion, i.e.,
27 videos were used for training and a summary of the
remaining video was generated for evaluation purposes. The
CNN for 2D pose estimation was trained independently prior
to each experiment, it was fine-tuned with the 27 training
videos in order to estimate the joints of the video used for
evaluation. This process was repeated for each video and for
each group E and NE, to result in 28 experienced summaries
and 28 inexperienced summaries. The generated summaries
had the same length as their respective ground truth. Figure 5
illustrates a few examples of the frames of a video as well as
highlight frames extracted by the proposed method (framed in
orange).
1) Objective evaluation by segment f-score: The ability of
the proposed method to extract highlights was evaluated in
terms of the f-score. In the proposed method, a one-second
period of video is as follows:
• true positive (TP), if it is in the summary and lt = 1,
• false positive (FP), if it is in the summary but lt = 0,
• false negative (FN), if it is not in the summary but lt = 1,
• true negative (TN), if it is not in the summary and lt = 0.
The f-score is subsequently defined as follows:
2TP
.
(6)
2TP + FP + FN
Table II shows the f-scores for the summaries generated with
the labels of both E and NE groups. In addition to the features
described in Section IV-A, it includes the results of using
the features from our previous work in UGSV summarization
[10]. The features were obtained by feeding the 3D body joint
representation of players’ actions to the action recognition
method in [55] and considering the action classification results.
Additionally, the proposed architecture was also compared
with that of the method used in the previous work [10]
that uses a hidden Markov model with Gaussian mixture
emission (GMM-HMM) over the same action recognition
results mentioned above. Finally, the results of using k-means
clustering are included, since k-means is widely accepted as
a baseline for user-generated video summarization [56]. To
f-score =

Body joint-based and holistic features
3D joints + CNN-ISA
2D joints + CNN-ISA
90 × 90
52 × 52
492 × 400
454 × 400
400 × 400
400 × 400
400 × 100
400 × 100
100 × 2
100 × 2

TABLE II
F- SCORE COMPARISON OF DIFFERENT COMBINATIONS OF FEATURES AND
OTHER UGSV SUMMARIZATION METHODS .
Method
Body joint-based
features
Holistic features
Body joint-based
and holistic features
Other UGSV
summarization

3D joints
2D joints
Action recognition [10]
CNN-ISA
C3D
3D joints + CNN-ISA
2D joints + CNN-ISA
GMM-HMM [10]
k-means clustering

Group E
0.53
0.45
0.48
0.50
0.27
0.58
0.57
0.44
0.28

Group NE
0.83
0.77
0.76
0.79
0.60
0.85
0.81
0.79
0.61

implement the k-means clustering baseline, the video segments
S were clustered based on the concatenated features 3D joints
+ CNN-ISA, and the summary was created by concatenating
in time the cluster centroids. The number of clusters for each
video were configured such that the resulting summary length
is equal to that of the ground truth.
With respect to using a single feature (i.e. 3D joins, 2D
joints, CNN-ISA, C3D, or action recognition), 3D joints obtain
the best performance. Although C3D features perform well
in action recognition tasks [11], the results were significantly
worse than that of other features in the summarization task.
The dimensionality of the C3D features is significantly higher
when compared to that of others, and thus the dataset may
not be sufficient to train the network well. Fine-tuning C3D
using the Kendo dataset might improve its performance. In
contrast, CNN-ISA also uses RGB frames and obtains better
results when compared to those of C3D and even 2D joints.
This implies that it is also possible to obtain features from
RGB frames that allow the modeling of UGSV highlights.
The decrease in the performance of 2D joints with respect to
3D joints may indicate that view variations in the same pose
negatively affect the body joint-based features stream. The
action recognition feature had an intermediate performance.
A potential reason is that the action recognition feature is
based on a classic approach for classification, so useful cues
contained in the 3D body joint positions degenerated in this
process. From these results, the features that performed better
for highlight classification correspond to CNN-ISA holistic
features and 3D body joint-based features.
Several state-of-the-art action recognition methods enjoy
improvements in performance by combining handcrafted
spatio-temporal features (e.g., dense trajectories) and those
learned via CNNs [11], [17]. This is also true in the present

8

𝑐=70%

𝑐 =0% 𝑐 =100%

Summary
Ground truth
Fig. 6. Association of highlights with respect to the greedy algorithm. Each
highlight in the ground truth is uniquely associated to a highlight in the
generated summary (two summary highlights cannot share the same ground
truth highlight). The completeness of a summary highlight corresponds to the
percentage of overlap with the ground truth (0% if unassociated).

work where a combination of CNN-ISA with 3D joints
achieves the best performance. The combination of CNNISA with 2D joints also provides a considerable boost in
performance and especially for the experienced summaries.
This supports our hypothesis that a two-streams architecture
also provides better results for UGSV summarization.
Finally, as shown in the lowest part of Table II, the results
of the proposed method outperform the results of previous
work, as well as those of the clustering-based baseline. While
clustering allows a wider variety of scenes in the summary,
this is not a good strategy for UGSV summarization that
follows a different criterion based on interestingness. The
proposed method also outperforms the previous work [10],
which used the classification results of an action recognition
method to train a GMM-HMM for highlight modeling. We
can conclude that it is not necessary to explicitly recognize
players’ actions for UGSV summarization, which may actually
degrade performance when compared to that in the case of
directly using action recognition features.
2) Objective evaluation by highlight completeness: A highlight may consist of consecutive video segments. Hence,
although missing a segment may not significantly impact the
f-score, it affects the continuity of the video, and thereby the
comprehensibility and the user experience of the summary.
Given this, a criterion is defined to evaluate the completeness
c of an extracted highlights as the fraction of overlap between
the extracted highlight and its associated ground truth highlights (Fig. 6). The association between extracted and ground
truth highlights is not trivial, and it was performed by using
a greedy algorithm in which the total c of all highlights is
maximized. An extracted highlight is considered as a TP if its
completeness c exceeds a certain percentage C%, and based
on this, the precision and recall of the highlights are calculated
as follows:
precision =

TP
,
TP + FP

recall =

TP
.
TP + FN

(7)

In the experiment, the threshold θ was varied from 0 to 1
over the probability p to generate the recall-precision curve of
group E and NE.
Figure 7 shows the curves produced for C = 50%, 70%,
and 90%. We observe that reducing C to 50% significantly
increases the number of complete highlights. The presence of
incomplete highlights is attributed to the way highlights are
extracted. First, the high p segments are extracted, and then the
highlight is completed with low p segments as the threshold
θ decreases (Fig. 8). However, prior to the completion of a

Fig. 7. Recall-precision curves for different completeness values (up: labels
E, down: labels NE). The gap between the curves C = 50% and C = 70%
shows that a significant number of the highlights are missing for a maximum
of half the interesting segments.

highlight, high p segments from other highlights are extracted
and, in a few cases, the low p segments are never extracted.
Specifically, the parts before and after an interesting Kendo
technique normally correspond to low p segments since they
are not present in every ground truth highlight annotated by
the participants.
The reason for the increased number of incomplete segments (less TP) in the NE summaries is because the inexperienced group annotated a higher number of highlights.
3) Subjective evaluation: The same participants who annotated the original videos were asked to participate in a survey
to assess their opinion on the ground truth and the generated
summaries. The three videos with the highest, median and
lowest f-scores (averaged over groups E and N E) were
selected. With respect to each video, participants were shown
the ground truth and the summaries generated with the best
feature combination (i.e., 3D joints + CNN-ISA) using both
group E and NE labels. As a result, each participant watched
12 videos (3 f-scores × 4 video types).
The participants were asked to:
• (Q1) assign a score in a Likert scale from 1 (very
few highlights are interesting) to 5 (most highlights are
interesting) based on their satisfaction with the contents
of each of the 12 videos.
• (Q2) state their opinion on the videos and the criteria
followed while assigning a score.
Table III shows the results of Q1 grouped by video type and
video f-score. The scores are averaged for group E and NE

9

𝑝
𝑝
𝜃

Ground
truth
Time

Fig. 8. Original length: 10 min 40 sec. Summary length: 1 min. The highlights summary is generated by applying a threshold θ to the probability of
interestingness p. Video segments with higher p are extracted prior to segments with lower p, and thus in a few cases the beginning/end segments of the
highlights are missing when compared to the ground truth.

separately.
In the context of Q1, with respect to the video type,
both experienced and inexperienced participants assigned a
higher score to the ground truth videos than to the generated
summaries. This is because some of the summaries contain
uninteresting video segments and also the completeness of
the highlights is worse when compared to that of ground
truth videos. The potential reasons as to why the ground truth
videos did not obtain a perfect score are mainly attributed to
the following two factors: (1) The ground truth summaries
are created by combining labels from several participants via
majority voting, and thus the original labels of each participant are lost. (2) The ground truth also contains incomplete
highlights due to errors when the participants annotated the
videos. Additionally, experienced participants preferred the NE
ground truth to the E summaries plausibly because they do not
find incomplete highlights interesting since context is missing.
Conversely, inexperienced participants tend to appreciate the
highlights from the experienced participants more than their
own highlights. This is potentially because the highlights from
the experienced participants are briefer and contain certain
techniques (e.g. counterattacks) that make summaries more
interesting when compared to those of the inexperienced
participants. The results for Q1 in terms of the f-score type
demonstrate the high correlation to the f-score (i.e., a video
with a higher f-score tends to receive a higher subjective
score).
With respect to Q2, participants provided their opinion on
the summaries. A few experienced participants found the highlights as too short and this even included complete highlights
in the ground truth. This occurs because only the segments
labeled as highlights by at least 40% of the participants (i.e., 2
people in group E and 4 people in group NE) were included in
the ground truth, and thus some labeled segments were left out.
Inexperienced participants state the usefulness of the proposed
method to extract highlights based on interesting actions as
well as time saved by watching the highlights as opposed to the
whole video. In addition, for a few inexperienced participants
incomplete highlights make the summaries difficult to follow.
From this evaluation, we conclude that the labels from

experienced users contain a better selection of Kendo techniques. Due to the negative impact of incomplete highlights
on the summaries, it is necessary to consider extra temporal
consistency in pt . Therefore, although combining the labels of
several participants is convenient to generate the ground truth,
the process introduces incomplete highlights (Section IV-B2)
and alters personal preferences. Thus, instead of combining
labels from different participants, another possibility is to
create personalized summaries with a higher quality ground
truth or to include user profiles such as that proposed in [5].

V. C ONCLUSION
This paper has described a novel method for automatic
summarization of UGSV, especially demonstrating the results for Kendo (Japanese fencing) videos. Given the lack
of editing conventions that permit the use of heuristics, a
different cue, i.e., players’ actions, is used to acquire high-level
semantics from videos to generate a summary of highlights.
The presented two-stream method combines body joint-based
features and holistic features for highlights extraction. The
best combination among the evaluated features corresponds
to a combination of 3D body joint-based features and CNNISA features [12]). In contrast to the previous work [10],
the results indicate that it is not necessary to explicitly
recognize players’ actions in order to determine highlights.
Alternatively, deep neural networks are leveraged to extract a
feature representation of players’ actions and to model their
temporal dependency. Specifically, LSTM is useful to model
the temporal dependencies of the joint positions of players’
bodies in each video segment as well as the highlights in
the entire video. In order to generate appealing summaries,
players’ 3D body joint positions from depth maps offer the
best performance. However, in the absence of depth maps, 2D
body joint positions and holistic features extracted from RGB
images are also used for summarization.
The future work includes fine-tuning the proposed network to illustrate its potential performance and evaluating the
method in the context of a wider variety of sports (e.g., boxing,
fencing, and table tennis).

10

TABLE III
S UBJECTIVE EVALUATION RESULTS WITH RESPECT TO THE VIDEO TYPE AND F - SCORE .
E ACH CELL CONTAINS THE MEAN ± THE STANDARD DEVIATION OF SCORES .

Group E
Group NE

Ground truth E
3.2±0.99
3.57±0.72

Video type
Ground truth NE
Summary E
3.07±1.04
2.6±1.23
3.5±1.07
3.2±0.83

ACKNOWLEDGMENT
This work was supported in part by JSPS KAKENHI Grant
Number 16K16086.
A PPENDIX A
Table IV lists the duration of the videos in the dataset used
in the experiments and their respective ground truth highlights
as annotated by users.
TABLE IV
D URATION OF THE VIDEO DATASET AND GROUND TRUTHS .
ID
#1
#2
#3
#4
#5
#6
#7
#8
#9
#10
#11
#12
#13
#14
#15
#16
#17
#18
#19
#20
#21
#22
#23
#24
#25
#26
#27
#28
Total

Original video
10 min 48 sec
5 min 10 sec
5 min 18 sec
9 min 37 sec
9 min 59 sec
10 min 5 sec
10 min 3 sec
10 min 10 sec
5 min 17 sec
5 min 14 sec
4 min 58 sec
20 min 40 sec
10 min 15 sec
10 min 16 sec
10 min 37 sec
10 min 37 sec
5 min 14 sec
5 min 4 sec
10 min 57 sec
5 min 36 sec
5 min 36 sec
10 min 48 sec
9 min 44 sec
10 min 23 sec
10 min 7 sec
10 min 40 sec
4 min 59 sec
8 min 13 sec
4 hours 6 min 11 sec

Ground truth E
1 min 11 sec
49 sec
1 min 9 sec
1 min 37 sec
2 min 33 sec
1 min 28 sec
48 sec
45 sec
32 sec
22 sec
53 sec
1 min 24 sec
53 sec
58 sec
47 sec
34 sec
16 sec
32 sec
38 sec
27 sec
33 sec
58 sec
1 min 11 sec
54 sec
28 sec
49 sec
33 sec
47 sec
24 min 49 sec

Ground truth NE
2 min 21 sec
1 min 7 sec
1 min 58 sec
2 min 17 sec
2 min 42 sec
2 min 55 sec
1 min 45 sec
2 min 14 sec
1 min 14 sec
1 min 30 sec
1 min 50 sec
4 min 14 sec
2 min 50 sec
5 min 8 sec
2 min 44 sec
2 min 21 sec
1 min 44 sec
2 min 21 sec
2 min 11 sec
1 min 21 sec
1 min 35 sec
1 min 59 sec
2 min 48 sec
2 min 25 sec
1 min 57 sec
2 min 5 sec
2 min 13 sec
2 min 10 sec
1 hour 3 min 59 sec

R EFERENCES
[1] Y. G. Jiang, Q. Dai, T. Mei, Y. Rui, and S. F. Chang, “Super fast
event recognition in internet videos,” IEEE Transactions on Multimedia,
vol. 17, no. 8, pp. 1174–1186, 2015.
[2] R. W. Lienhart, “Dynamic video summarization of home video,” in Proc.
SPIE Electronic Imaging, 1999, pp. 378–389.
[3] M. Otani, Y. Nakashima, E. Rahtu, J. Heikkilä, and N. Yokoya, “Video
summarization using deep semantic features,” in Proc. Asian Conference
on Computer Vision, 2016, pp. 361–377.
[4] W. Li, Z. Zhang, and Z. Liu, “Action recognition based on a bag of
3D points,” in Proc. IEEE Conference on Computer Vision and Pattern
Recognition, 2010, pp. 9–14.
[5] N. Nitta, Y. Takahashi, and N. Babaguchi, “Automatic personalized
video abstraction for sports videos using metadata,” Multimedia Tools
and Applications, vol. 41, no. 1, pp. 1–25, 2009.

Summary NE
2.73±0.87
2.9±0.97

Highest
3.3±0.95
3.48±0.83

Video f-score
Median
Lowest
2.85±0.97
2.55±1.18
3.03±0.91
3.38±0.95

[6] H. Pan, P. Van-Beek, and M. I. Sezan, “Detection of slow-motion replay
segments in sports video for highlights generation,” in Proc. IEEE
International Conference on Acoustics, Speech, and Signal Processing,
vol. 3, 2001, pp. 1649–1652.
[7] A. Ekin, A. M. Tekalp, and R. Mehrotra, “Automatic soccer video
analysis and summarization,” IEEE Transactions on Image Processing,
vol. 12, no. 7, pp. 796–807, 2003.
[8] G. Xu, Y. F. Ma, H. J. Zhang, and S. Q. Yang, “An HMM-based
framework for video semantic analysis,” IEEE Transactions on Circuits
and Systems for Video Technology, vol. 15, no. 11, pp. 1422–1433, 2005.
[9] B. Li and M. I. Sezan, “Event detection and summarization in sports
video,” in Proc. IEEE Workshop on Content-based Access of Image and
Video Libraries, 2001, pp. 132–138.
[10] A. Tejero-de-Pablos, Y. Nakashima, T. Sato, and N. Yokoya, “Human
action recognition-based video summarization for RGB-D personal
sports video,” in Proc. IEEE International Conference on Multimedia
and Expo, 2016, pp. 1–6.
[11] D. Tran, L. Bourdev, R. Fergus, L. Torresani, and M. Paluri, “Learning
spatiotemporal features with 3d convolutional networks,” in 2015 IEEE
International Conference on Computer Vision, 2015, pp. 4489–4497.
[12] Q. V. Le, W. Y. Zou, S. Y. Yeung, and A. Y. Ng, “Learning hierarchical
invariant spatio-temporal features for action recognition with independent subspace analysis,” in Proc. IEEE Conference on Computer Vision
and Pattern Recognition, 2011, pp. 3361–3368.
[13] D. Y. Chen, S. W. Shih, and H. Y. M. Liao, “Human action recognition
using 2-D spatio-temporal templates,” in Proc. IEEE International
Conference on Multimedia and Expo, 2007, pp. 667–670.
[14] H. Meng, N. Pears, and C. Bailey, “A human action recognition system
for embedded computer vision application,” in Proc. IEEE Conference
on Computer Vision and Pattern Recognition, 2007, pp. 1–6.
[15] S. Calderara, R. Cucchiara, and A. Prati, “Action signature: A novel
holistic representation for action recognition,” in Proc. IEEE International Conference on Advanced Video and Signal Based Surveillance,
2008, pp. 121–128.
[16] K. Simonyan and A. Zisserman, “Two-stream convolutional networks for
action recognition in videos,” in Proc. Advances in Neural Information
Processing Systems, 2014, pp. 568–576.
[17] C. Feichtenhofer, A. Pinz, and R. Wildes, “Spatiotemporal residual networks for video action recognition,” in Proc. Conference and Workshop
on Neural Information Processing Systems, 2016, pp. 3468–3476.
[18] M. A. Goodale and A. D. Milner, “Separate visual pathways for
perception and action,” Trends in neurosciences, vol. 15, no. 1, pp. 20–
25, 1992.
[19] L. Xia, C. C. Chen, and J. K. Aggarwal, “View invariant human action
recognition using histograms of 3D joints,” in Proc. IEEE Conference
on Computer Vision and Pattern Recognition, 2012, pp. 20–27.
[20] M. Martı́nez-Zarzuela, F. J. Dı́az-Pernas, A. Tejero-de-Pablos,
D. González-Ortega, and M. Antón-Rodrı́guez, “Action recognition
system based on human body tracking with depth images,” Advances
in Computer Science: an International Journal, vol. 3, no. 1, pp. 115–
123, 2014.
[21] X. Cai, W. Zhou, L. Wu, J. Luo, and H. Li, “Effective active skeleton
representation for low latency human action recognition,” IEEE Transactions on Multimedia, vol. 18, no. 2, pp. 141–154, 2016.
[22] J. Choi, W. J. Jeon, and S. C. Lee, “Spatio-temporal pyramid matching
for sports videos,” in Proc. ACM International Conference on Multimedia Information Retrieval, 2008, pp. 291–297.
[23] M. Chen, S. C. Chen, M. L. Shyu, and K. Wickramaratna, “Semantic
event detection via multimodal data mining,” IEEE Signal Processing
Magazine, vol. 23, no. 2, pp. 38–46, 2006.
[24] D. Tjondronegoro, Y. P. Chen, and B. Pham, “Integrating highlights for
more complete sports video summarization,” IEEE MultiMedia, vol. 11,
no. 4, pp. 22–37, 2004.

11

[25] C. Liang, J. Kuo, W. Chu, and J. Wu, “Semantic units detection and
summarization of baseball videos,” in Proc. IEEE Midwest Symposium
on Circuits and Systems, vol. 1, 2004, pp. 297–300.
[26] A. Divakaran, K. A. Peker, R. Radhakrishnan, Z. Xiong, and R. Cabasson, “Video summarization using mpeg-7 motion activity and audio
descriptors,” in Video Mining. Boston, MA: Springer US, 2003, pp.
91–121.
[27] C. Y. Chen, J. C. Wang, J. F. Wang, and Y. H. Hu, “Motion entropy
feature and its applications to event-based segmentation of sports video,”
EURASIP Journal on Advances in Signal Processing, vol. 2008, pp. 1–8,
2008.
[28] E. Mendi, H. B. Clemente, and C. Bayrak, “Sports video summarization
based on motion analysis,” Computers & Electrical Engineering, vol. 39,
no. 3, pp. 790–796, 2013.
[29] X. S. Hua, L. Lu, and H. J. Zhang, “Optimization-based automated home
video editing system,” IEEE Transactions on Circuits and Systems for
Video Technology, vol. 14, no. 5, pp. 572–583, 2004.
[30] M. Mills, J. Cohen, and Y. Y. Wong, “A magnifier tool for video data,”
in Proc. ACM SIGCHI Conference on Human Factors in Computing
Systems, 1992, pp. 93–98.
[31] S. Lu, Z. Wang, T. Mei, G. Guan, and D. D. Feng, “A bag-of-importance
model with locality-constrained coding based feature learning for video
summarization,” IEEE Transactions on Multimedia, vol. 16, no. 6, pp.
1497–1509, 2014.
[32] Y. F. Ma, X. S. Hua, L. Lu, and H. Zhang, “A generic framework of
user attention model and its application in video summarization,” IEEE
Transactions on Multimedia, vol. 7, no. 5, pp. 907–919, 2005.
[33] G. Evangelopoulos, A. Zlatintsi, A. Potamianos, P. Maragos, K. Rapantzikos, G. Skoumas, and Y. Avrithis, “Multimodal saliency and fusion
for movie summarization based on aural, visual, and textual attention,”
IEEE Transactions on Multimedia, vol. 15, no. 7, pp. 1553–1568, 2013.
[34] W. T. Peng, W. T. Chu, C. H. Chang, C. N. Chou, W. J. Huang, W. Y.
Chang, and Y. P. Hung, “Editing by viewing: automatic home video
summarization by viewing behavior analysis,” IEEE Transactions on
Multimedia, vol. 13, no. 3, pp. 539–550, 2011.
[35] H. Yang, B. Wang, S. Lin, D. Wipf, M. Guo, and B. Guo, “Unsupervised
extraction of video highlights via robust recurrent auto-encoders,” in
Proc. IEEE International Conference on Computer Vision, 2015, pp.
4633–4641.
[36] M. Otani, Y. Nakashima, T. Sato, and N. Yokoya, “Video summarization
using textual descriptions for authoring video blogs,” Multimedia Tools
and Applications, vol. 76, no. 9, pp. 12 097–12 115, 2017.
[37] Y. Song, J. Vallmitjana, A. Stent, and A. Jaimes, “Tvsum: Summarizing
web vvideos using titles,” in Proc. IEEE Conference on Computer Vision
and Pattern Recognition, 2015, pp. 5179–5187.
[38] K. Simonyan and A. Zisserman, “Very deep convolutional networks for
large-scale image recognition,” in Proc. International Conference on
Learning Representations, 2015.
[39] C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan,
V. Vanhoucke, and A. Rabinovich, “Going deeper with convolutions,”
in Proc. IEEE Conference on Computer Vision and Pattern Recognition,
2015, pp. 1–9.
[40] Z. Zhang, “Microsoft Kinect sensor and its effect,” IEEE MultiMedia,
vol. 19, no. 2, pp. 4–10, 2012.
[41] S. E. Wei, V. Ramakrishna, T. Kanade, and Y. Sheikh, “Convolutional
pose machines,” in Proc. IEEE Conference on Computer Vision and
Pattern Recognition, 2016, pp. 3714–3722.
[42] A. Karpathy, G. Toderici, S. Shetty, T. Leung, R. Sukthankar, and
L. Fei-Fei, “Large-scale video classification with convolutional neural
networks,” in Proc. IEEE Conference on Computer Vision and Pattern
Recognition, 2014, pp. 1725–1732.
[43] K. H. Zeng, T. H. Chen, J. C. Niebles, and M. Sun, “Title generation
for user generated videos,” in Proc. European Conference on Computer
Vision, 2016, pp. 609–625.
[44] Y. Jia, E. Shelhamer, J. Donahue, S. Karayev, J. Long, R. Girshick,
S. Guadarrama, and T. Darrell, “Caffe: Convolutional architecture for
fast feature embedding,” in Proc. ACM International Conference on
Multimedia, 2014, pp. 675–678.
[45] J. Liu, J. Luo, and M. Shah, “Recognizing realistic actions from videos
“in the wild”,” in Proc. IEEE Conference on Computer Vision and
Pattern Recognition, 2009, pp. 1996–2003.
[46] M. Marszalek, I. Laptev, and C. Schmid, “Actions in context,” in Proc.
IEEE Conference on Computer Vision and Pattern Recognition, 2009,
pp. 2929–2936.
[47] M. D. Rodriguez, J. Ahmed, and M. Shah, “Action MACH a spatiotemporal Maximum Average Correlation Height filter for action recog-

[48]
[49]

[50]
[51]
[52]
[53]
[54]
[55]

[56]

nition,” in Proc. IEEE Conference on Computer Vision and Pattern
Recognition, 2008, pp. 1–8.
K. Soomro, A. R. Zamir, and M. Shah, “UCF101: A dataset of 101
human actions classes from videos in the wild,” CRCV-TR-12-01, 2012.
J. Y.-H. Ng, M. Hausknecht, S. Vijayanarasimhan, O. Vinyals, R. Monga,
and G. Toderici, “Beyond short snippets: Deep networks for video
classification,” in Proc. IEEE Conference on Computer Vision and
Pattern Recognition, 2015, pp. 4694–4702.
B. T. Truong and S. Venkatesh, “Video abstraction: A systematic
review and classification,” ACM Transactions on Multimedia Computing,
Communications, and Applications, vol. 3, no. 1, pp. 3:1–3:37, 2007.
M. Linna, J. Kannala, and E. Rahtu, “Real-time human pose estimation
from video with convolutional neural networks,” CoRR, abs/1609.07420,
2016.
H. Wang, M. M. Ullah, A. Klaser, I. Laptev, and C. Schmid, “Evaluation
of local spatio-temporal features for action recognition,” in Proc. British
Machine Vision Conference, 2009, pp. 124–1.
S. Tokui, K. Oono, S. Hido, and J. Clayton, “Chainer: A next-generation
open source framework for deep learning,” in Proc. Conference and
Workshop on Neural Information Processing Systems, 2015, pp. 1–6.
D. P. Kingma and J. L. Ba, “Adam: a method for stochastic optimization,” in Proc. International Conference on Learning Representations,
2015, pp. 1–13.
A. Tejero-de-Pablos, Y. Nakashima, N. Yokoya, F. J. Dı́az-Pernas, and
M. Martı́nez-Zarzuela, “Flexible human action recognition in depth
video sequences using masked joint trajectories,” EURASIP Journal on
Image and Video Processing, vol. 2016, no. 1, pp. 1–12, 2016.
Y. Cong, J. Yuan, and J. Luo, “Towards scalable summarization of
consumer videos via sparse dictionary selection,” IEEE Transactions
on Multimedia, vol. 14, no. 1, pp. 66–75, 2012.

Antonio Tejero-de-Pablos received a B.E. in telematics engineering and a M.E.in telecommunications
engineering from University of Valladolid, Spain, in
2009 and 2012 respectively. He received a Ph.D. in
information sciences from Nara Institute of Science
and Technology (NAIST), Japan, in 2017. He is
currently a researcher at the University of Tokyo. His
research interests include action recognition, video
summarization, user interfaces and general-purpose
computing on GPU. He is a member of the IEEE.

Yuta Nakashima received a B.E. and a M.E. in
communication engineering and a Ph.D. in engineering from Osaka University, Osaka, Japan in
2006, 2008, and 2012, respectively. He was an
assistant professor at Nara Institute of Science and
Technology from 2012 to 2016 and is currently
an associate professor at the Institute for Datability
Science, Osaka University. He was a visiting scholar
at UNCC in 2012 and at CMU from 2015-2016.
His research interests include computer vision and
machine learning and their applications. His main
research focus includes video content analysis using machine learning approaches. He is a member of IEEE, ACM, IEICE, and IPSJ.

Tomokazu Sato received a B.E. in computer and
system science from Osaka Prefecture University in
1999. He received a M.E. and a Ph.D. in information
sciences from Nara Institute of Science and Technology (NAIST) in 2001 and 2003, respectively. He
was an assistant professor at NAIST during 20032011. He served as a visiting researcher at Czech
Technical University in Prague from 2010-2011. He
is an associate professor at NAIST since 2011. His
research interests include computer vision and mixed
reality.

12

Naokazu Yokoya received a B.E., a M.E., and a
Ph.D. in information and computer sciences from
Osaka University in 1974, 1976, and 1979, respectively. He joined the Electrotechnical Laboratory
(ETL) at the Ministry of International Trade and Industry in 1979. He was a visiting professor at McGill
University in 1986-1987 and was a professor at
Nara Institute of Science and Technology (NAIST)
during 1992-2017. He is the president at NAIST
since April 2017. His research interests include
image processing, computer vision, and mixed and
augmented reality. He is a life senior member of IEEE.

Marko Linna received a M.Sc. in computer science
and engineering from the University of Oulu, Finland, in 2016. He is a research assistant at the Center
for Machine Vision and Signal Analysis (CMVS)
research center at the University of Oulu. His main
research interests include computer vision.

Esa Rahtu received a MSc in 2004 and a Doctor
of Technology in electrical engineering in 2007
from the University of Oulu, Finland. His thesis
was selected as the best doctoral thesis by the
Pattern Recognition Society of Finland. He was also
awarded a highly competitive three-year postdoctoral
scholarship from the Academy of Finland. Currently,
he is an assistant professor at the Tampere University
of Technology in Finland. His main research interests include computer vision and machine learning
with a focus on features extraction, object classification and categorization, face recognition, image retrieval, and registration
and segmentation of deformable scenes.

