Multi-Level Recurrent Residual Networks for Action Recognition
1

Zhenxing ZHENG1,2
Gaoyun AN1,2
Qiuqi RUAN1,2
Institute of Information Science, Beijing Jiaotong University, Beijing 100044, China
2
Beijing Key Laboratory of Advanced Information Science
and Network Technology, Beijing 100044, China
{zhxzheng, gyan, qqruan}@bjtu.edu.cn

arXiv:1711.08238v6 [cs.CV] 3 Jan 2018

Abstract
Most existing Convolutional Neural Networks(CNNs) used for action recognition are either
difficult to optimize or underuse crucial temporal
information. Inspired by the fact that the recurrent
model consistently makes breakthroughs in the task
related to sequence, we propose a novel Multi-Level
Recurrent Residual Networks(MRRN) which incorporates three recognition streams. Each stream consists
of a Residual Networks(ResNets) and a recurrent
model. The proposed model captures spatiotemporal
information by employing both alternative ResNets to
learn spatial representations from static frames and
stacked Simple Recurrent Units(SRUs) to model temporal dynamics. Three distinct-level streams learned
low-, mid-, high-level representations independently
are fused by computing a weighted average of their
softmax scores to obtain the complementary representations of the video. Unlike previous models which
boost performance at the cost of time complexity and
space complexity, our models have a lower complexity
by employing shortcut connection and are trained
end-to-end with greater efficiency. MRRN displays
significant performance improvements compared to
CNN-RNN framework baselines and obtains comparable performance with the state-of-the-art, achieving
51.3% on HMDB-51 dataset and 81.9% on UCF-101
dataset although no additional data.

Figure 1: CNN-RNN framework. A generic model,
CNN with Recurrent Neural Networks(RNN) for action
recognition. CNN is applied to encode the appearance of
video frames into a group of fixed length vectors, frame by
frame, that lately is decoded through RNN to learn video
representation spatially and temporally. Then, time series
pooling is used over outputs at all time-steps followed by a
softmax layer to predict activity.

components(e.g., human body [5], object [6], scene [7])
appeared in the image, and low-level features, including scale invariant feature transform(SIFT) [8] and Histogram of oriented gradients(HOGs) [9], are pursued
by researchers for the sake of characterizing actions.
Given a video sequence, it is natural to decompose
into spatial and temporal components. Compared with
the traditional still image based action recognition,
1 Introduction
videos provide additional temporal information for disWith the development of deep learning and the im- tinguishing behaviors. To utilize temporal information,
provement in computer hardware, action recognition several hand-crafted features such as Space Time Interattracts growing attention in the research community est Points(STIP) [10], dense trajectories with motion
[1–3]. There are many potential applications of action boundary histograms(MBH) Descriptors [11], SURF
recognition, like video caption, abnormal event detec- descriptors with the dense optical flow [12] are widely
tion, intelligent monitoring, and auto drive. However, used in action field on account of needing no algorithm
action recognition remains a fundamental challenge in to detect human body. They usually detect discriminacomputer vision, since it is affected by rapid move- tive regions for action analysis first and subsequently
ment, illumination variation, occlusion and viewpoint encode these local features into one vector as the reprevariation largely.
sentation of the image which is used to learn classifiers
In the early study, researchers mainly focus their at- for action recognition. Among these local features, imtention on the visual representation of the static image proved trajectories [12] achieves the best performance
which contains no motion information. The survey on on several challenging benchmarks(e.g., HMDB-51 [13],
still image [4] shows high-level cues, including visual Hollywood2 [14]) enormously.
1

Recently, Convolutional Neural Networks(CNNs)
[15] has been demonstrated as an effective way to automatically learn substantial discriminative visual representations and make significant breakthroughs in image classification, especially deep CNNs [16, 17]. Inspired by this, many works begin to generalize deep
learning methods to the field of action recognition.
To learn stronger spatiotemporal representations, 3D
convolutional networks [3], Trajectory-Pooled DeepConvolutional Descriptors [18], and LRCN [19] learn
spatiotemporal representations directly. Unlike these,
convolutional Two-Stream Networks [20], Multi-region
two-stream R-CNN [21] and cLSTM [22] use two networks to learn spatial and temporal representations independently and then fuse them. The learned temporal representations explicitly from optical flow maps by
Two-Stream Networks turn out to be better than C3D
relied on motion-sensitive convolutional kernels.

ments compared to CNN-RNN framework baselines
and obtains comparable performance with the stateof-the-art.
The rest of the paper is organized as follows. In
section 2, we review various state-of-the-art methods
addressing partly challenging problems in action recognition. The method we proposed in this paper is described in section 3. Implement details are introduced
in section 4. We make analyses on the experimental
results in section 5. Finally, we draw conclusions.

2

Related Work

Hand-crafted features. Local features are of popularity in image classification, which characterizes images through descriptors such as SIFT, HOGs, and
SURF. Image-based action recognition concentrates on
identifying actions appeared in the static image. Exemplarlet [5] comprises abundant visual information
(e.g., pose) within the body for discerning human actions. For this purpose, what need to do is manually
selecting and segmenting bounding box in images. Objectness method [6] quantifies the probability of the
bounding box that contains the object of any class,
so multiple candidates relevant to actions (e.g., bike,
basketball, laptop) can be found from the cluttered
background. The integrative model [7] integrates scene
and object categorizations to discriminate events occurred in images. Nevertheless, low-level features usually can’t work well alone due to the cluttered background and crucial temporal information discarded by
these methods. In order to address above problems,
previous research directly extends the recognition algorithms based on the static image to learn spatiotemporal representations. Traditional video-based action
recognition [23] is described by a collection of local descriptors. For example, extended from the Harris corner detector [24], Harris3D detector [10] was proposed
to encode the region of the interest(ROI). RMM [25]
encodes the layout of hybrid features for action discrimination. Unlike this, Dense trajectories [11] tracks
dense points that are sampled in each frame depending
on the dense optical flow field. This method is shown
to be high capacity for video representations. Based
on this work, Improved dense trajectories [12] takes
camera motion into account and removes the trajectories which are consistent with the camera, achieving state-of-the-art results. Whereas, extracting handcrafted features along the trajectory have higher computational complexity.
Deep-learning features. Convolutional Networks
has shown it can extract deep spatial features for image classification. Encouraged by the impressive performance, many works make extensive attempts at using deep-learning methods for action recognition. The
prior attempt is the strategies [1] of fusing information
across the time domain such as late fusion, early fusion or slow fusion so that high layers acquire plentiful
spatiotemporal information. 3D Convolutional Neu-

Despite good performance, these methods are either
computationally intensive or insufficient in space domain. To deal with these, we propose a novel MultiLevel Recurrent Residual Networks(MRRN) model for
action recognition, as illustrated in fig.2. In the proposed model, ResNets is applied to encode the appearance of video frames into fixed length vectors that
lately are decoded through stacked SRUs to produce
video representations. Employing identity shortcut
connections in MRRN is parameter-effective, which
lowers space complexity and time complexity by a large
margin. Then, time series pooling is used over the entire outputs at all time-steps and predict the probability score by softmax layer. Considering the fact that
higher activation values of different depths in the network gather around different parts of the image, for
example, mid-level representations focus on legs while
high-level representations focus on the whole body, we
develope three different-level models to produce complementary representations and make final predictions.
In MRRN, three separate sub-models are called low, mid-, high-level Recurrent Residual Networks(RRN)
respectively.
The contributions of this paper are shown as follows:
First, we analyze the effect of diverse hyper-parameter
settings qualitatively to illustrate the general tendency
of performance. To lower the space complexity and
time complexity, we propose to use identity shortcut
connections in the proposed model. Additionally, we
experiment with low-, mid-, high-level representations
of the video in various time pooling manners, experimentally demonstrating how well different-level features contribute to action recognition. Based on the
complementary relation between the whole and the
part, our proposed architecture consists of three separate recognition streams captured different-level information to learn effective representations for action
recognition. Finally, a series of experiments were carried out on two standard video actions benchmarks of
HMDB-51 and UCF-101 dataset. Experimental results
show MRRN displays significant performance improve2

Figure 2: Pipline of MRRN. Our network consists of three independent models. We capitalize on different-level
RRN to produce different-level video representations simultaneously. Then these representations are fused by computing
a weighted average of their softmax scores to obtain the complementary representations for action recognition.
used to extract different-level representations of the
video sequence. To form the compositional representation in time domain, stacked SRUs [28] processes hierarchical features further followed by a pooling layer
and a softmax layer for predicting the activity. In the
end, three different-level models are combined by late
fusion.

ral Networks(3D CNN) directly captures spatiotemporal information from multiple adjacent frames by
replacing the 2 × 2 convolutional kernel with 3 × 3.
Beyond that, an alternative way of associating RGB
appearance with motion is Two-Stream Convolutional
Networks [2]. This model contains two identical networks where one net for RGB images and another
for stacked optical flow images. Two networks are
trained independently and then combined by using average or linear SVM methods to fuse softmax scores
of each. Recently, considering the Long Short-Term
Memory(LSTM) playing a vital role in the task related
to sequence, CNN with LSTM [26] becomes an active
research topic in action recognition. Specifically, the
feature sequences are extracted by CNN from video
frames and then passed into LSTM step by step for
action recognition, as illustrated in fig.1.
Motivated by above analysis, we proposed a
novel model, Multi-Level Recurrent Residual Networks(MRRN), which integrates three distinct-level
ResNet-34 with stacked SRUs and use complementary
representations to classify actions. The details will be
introduced in the next section.

3.1

Residual Networks

Deep architectures [17, 29] play an important role
in ImageNet Large-Scale Visual Recognition Challenge2014 (ILSVRC2014) [30] and reveal that adding
network depth is of importance for improving performance and enriches hierarchical features. On the other
hand, the deep system is difficult to optimize on account of notorious vanishing and exploding gradients
[31] which impede convergence. In addition, accuracy
plateaus even degrades.
Recently, several novel networks with more than
100 layers are trained well by adding shortcut connection [32] or dense connection [33] to mitigate overfitting. These deep networks set a series of records
of highly challenging object recognition and classification benchmarks. Note that attention maps [34] show
3 Method
higher activation values of different depths in networks
In the proposed MRRN model, different-level gather around different parts of the image, and we subResNet-34 which was pre-trained on the challenging sequently investigate how well diverse representations
image classification dataset called ImageNet [27] is work for taking activity predictions.
3

The Residual Networks(ResNets) constructs an extremely deep network by formulating a desired residual mapping as Φ(x) and fitting another mapping of
F (x) = Φ(x) − x for each stacked layers called the
building block. Each block contains two layers or three
layers according to the requirement of depth, where two
layers are 3 × 3, 3 × 3 convolutional kernels and three
layers are 1 × 1, 3 × 3 and 1 × 1 convolutional kernels.
Between weight layers, rectification (ReLU) activation
function [35] is used to introduce nonlinearity. Employing 1 × 1 convolutional kernels can reduce time complexity while retaining similar space complexity. The
degrade and optimization problem with the increase of
the depth are addressed by learning residual functions
with reference to the layer inputs. In general, ResNets
consists of multiple residual blocks and perform short
connection (identity mapping in this case) from the
bottom to the top in each block, where information
flows from shallow to deep.
In this paper, we use ResNet-34 to build different
level representations. We choose the output activations of last three groups of residual blocks as low-,
mid, high-level representation depicted as Al , Am , Ah
and name corresponding ResNet-34 with different layers low-, mid-, high-level ResNet-34 respectively. Then
these appearance representations are pushed into SRUs
to obtain spatiotemporal video representations.
To represent the structure of ResNet-34, we use
shorthand notations expressed as follows: Conv2d,
BatchNorm2d, ReLU, MaxPool2d, First Sequential
BasicBlock, Second Sequential BasicBlock, Third Sequential BasicBlock, Fourth Sequential BasicBlock,
AvePool2d and Linear, where conv2d is 2D convolution, BatchNorm2d is 2D batchnormalization, ReLU is
rectified linear units, MaxPool2d is 2D max pooling,
Sequential BasicBlock is a group of building blocks,
AvePool2d is 2D average pooling and Linear is fully
connection. The last three Sequential BasicBlock
and its corresponding output activation tensor A ∈
RC×H×W can be represented as :

Figure 3: SRU Architecture. ψ represents the operation of 1 − input, g is the hyperbolic tangent function, σ is
the sigmoid function.
and ⊕ are element-wise multiplication and addition respectively. We did not show the bias
for simplication.

where Statei,t denotes input gate state, hidden gate
state and forget gate state at time t. It is apparent that
hidden state ht−1 in the last time is used for updating
all gate states at this time.
Different from the previous methods, we use the recurrent model proposed by [28] to capture temporal
information. The advantages of using SRUs for modeling temporal dynamics are three folds: First, it would
boost performance if substitute LSTM in some case.
Second, SRUs operates faster than LSTM. Third, deep
SRUs could be trained well by adding skip connection.
The SRUs architecture used in the proposed MRRN
is defined as follows:
x
et = W xt

(3)

ft = σ(Wf xt + bf )

(4)

rt = σ(Wr xt + br )

(5)

ct = ft

ct−1 + (1 − ft )

x
et

(6)

ht = rt

g(ct ) + (1 − rt )

xt

(7)

where ft , rt are sigmoid gates referred to the forget gate
and reset gate, g(·) is the hyperbolic tangent function,
Alevel = [Al1 , Al2 , ..., AlN ]
Al,i ∈ RC
(1) as illustrate in fig.3. SRUs breaks the dependency by
completely dropping ht−1 in the recursion, which simwhere level ∈ [low, mid, high] and N = H × W . plifies the state computation and discloses more paralSpecifically, Al ∈ R128×28×28 , Am ∈ R256×14×14 , lelism while retaining the strong capability for repreAh ∈ R512×7×7 . We then average these activations sentations. It is worth to mention that the update of
A ∈ RC×H×W and result in descriptors xf eature ∈ RC internal states ct still depends on the previous states
of video frames.
ct−1 . When input vectors xt are passed into SRUs
module, x
et , ft , Rt at each time-step are computed at
once.
Eq.6
and Eq.7 operations are elementwise.
3.2 Recurrent Model
As mentioned above, we average the activation tenRNNs, especially LSTM [36] are widely used in ma- sor:
N
chine translation, image caption and speech recogni1 X t
x
=
A
(8)
t
tion, achieving desirable results. The update of gate
N i=1 l,i
states in the recursion, however, depends on the previous hidden states ht−1 which greatly alleviates the as a fixed-length feature vector of f ramet and put it
calculation speed. The computations of gate states are into SRUs at time-step t resulted in the representation
rt . For fusing predictions at all time-steps, we employ
shown as follows:
both mean pooling and max pooling to obtain multiple
Statei,t = σ(Wxi xt + Whi ht−1 + bi )
(2) types of video representations.
4

Thus, our model consists of two phrases, see fig2. both datasets are used to find the better manner of
In the CNN phrases, we encode N continual video pooling and explore the secret of different-level repreframes belonged to one video as a feature sequence sentations.
X = (x1 , ..., xN ), where xt ∈ RC (C ∈ [128, 256, 512])
and in the recurrent phrase, the probability distribu- 4.2 Implement details
tion over action categories is calculated by the following
In the experiment of hyper-parameters, high-level
equations:
ResNet-34
was used to extract frame representations
PN
exp( N1 t=1 Wjt rt )
resulted in 512-dimensional feature vectors. For inves(9) tigating hyper-parameters, we experimented with the
Pmean (y = j) = PM
PN
1
t=1 Wjt rt )
j=1 exp( N
number of hidden units and set it to 256, 512 or 1024
exp(max∀t⊆[1,N ] Wjt rt )
Pmax (y = j) = PM
(10) units. Similarity, the number of layers was set to 3,
4 or 5 layers. While in the experiment of hierarchiexp(max∀
W
r
)
t⊆[1,N ] jt t
j=1
cal features, the setting of input to SRUs is consiswhere Wjt represents the weight parameters mapping tent with the shape of activation tensors produced by
inputs of the recurrent model at time t to action j, M ResNets. For low-, mid-, high-level frame representais the number of hidden units. Eq.9 and Eq.10 denote tions, the input size of the recurrent model is 128, 256
mean pooling prediction and max pooling prediction and 512 respectively. Nonlinear transformation in rerespectively. Finally, we use the following formula to current computations used sigmoid activation function
combine different-level representations and make the and we added the hyperbolic tangent activate layers on
final prediction.
the top of stacked SRUs for predictions at each timestep. Max pooling and mean pooling were performed
Pf inal = a × PH + b × PM + c × PL
(11) over all time-steps. It is worth to mention that we did
not adjust any structures of original ResNet-34 and rewhere P(·) in the left of the equation refers to the pretrain it to extract frame representations.
diction of different-level models and Pf inal is the final
The orthogonal weight initialization introduced by
prediction produced by combined model. According
[38] was used to initialize SRUs and all models were
to the performance of three different-level RRNs, we
trained from scratch in an end to end scheme. Adam
assign 0.7, 0.2 and 0.1 as a, b and c respectively in
optimizer algorithm [39] and CrossEntropyLoss funcfollowing experiments.
tion were used to optimize all models with mini-batch
size 28 for 12 epochs over the entire datasets. The initial learning rate was set to 1e-5 for the first 8 epochs
4 Experiments
and changed to 1e-6 for the rest. We initially tried
In this section, we first introduce two popular chal- to set learning rate 1e-4 which led to the convergence
lenging datasets, HMDB-51 dataset and UCF-101 quickly at first but obtained the relatively poor perfordataset [37]. Then we specify implement details of all mance in the end. We adopt the dropout regularization
ratio 0.5 for linear transformations to improve the genexperiments involved in this paper.
eralization of models.
The setting of our experiments followed the practice
4.1 Dataset
in [26]. The smaller side of the image was scaled to
UCF-101 dataset and HMDB-51 dataset are chal- 256 and then a 224 × 224 region was randomly cropped
lenging action recognition benchmarks because of lim- from the rescaled image or its horizon flip with 50%
ited data. UCF-101 dataset has 13320 videos that probability for data augmentation. Besides, the values
are collected from YouTuBe and organized as 101 ac- of image pixel were transformed from H × W × C in
tion categories. Every kind of behavior was conducted the range [0, 255] to the tensor of the shape C ×H ×W
by 25 humans and everyone did more than one times in the range [0, 1.0] and we normalized each channel
from 4 to 7. The action categories can be divided into in the way of channel = (channel − mean)/std with
five types: Human-Object Interaction, Body-Motion mean(0.485, 0.456, 0.406) and std(0.229, 0.224, 0.225).
Only, Human-Human Interaction, Playing Musical In- The same pre-process approaches were used in training
and testing except that we used randomly crop from
struments and Sports.
Moreover, the HMDB-51 dataset is collected from rescaled image in training while center crop in testing
various sources, mostly from movies. This dataset without horizon flip.
contains 3570 training clips and 1530 testing clips beVideos were split into 30 frames clips with a stride
longed to 51 distinct categories. These actions can be of 8 frames and the maximum of clips split from one
roughly divided into five groups, general facial actions, video was set to 20. We would loop the video if necfacial actions with object manipulation, general body essary when the length of videos is insufficient to 30
movements, body movements with object interaction frames. Thus, we obtained 21147 clips for training that
and body movements for human interaction. We use each served as an individual training sample and 8791
HMDB-51 dataset to illustrate the general relations clips for testing. In testing, we fused all predictions of
between hyper-parameter and performance. Besides, clips belonged to one video by averaging their softmax
5

Figure 4: Predictions on HMDB-51. Under each sub-picture, purple, green and red bars indicate the ground
truth label, correct prediction, and incorrect prediction respectively sorted in decreasing confidence. The length of bars
is on behalf of the probability of corresponding actions. The first row shows correct predictions and the second shows
wrong predictions.

scores.
Table 1: Experiments on Hyper-parameters
Depending on experiments in this paper, we select
setting
3layers 4layers 5layers
the best configuration and pooling manner as default
256
hiddenunits
48.10
48.10
43.92
settings to evaluate the effectiveness of MRRN. Here,
512
hiddenunits
48.17
49.28
46.21
we design three separate models according to low-,
1024 hiddenunits
50.78
49.61
47.71
mid- and high-level representations and are combined
by averaging their weighted softmax scores, as illustrated in fig.2.
All experiments were carried out on the first split of
Table 1 reports various experimental accuracies unHMDB-51 dataset or UCF-101 dataset and performed der different settings. The results from Table 1 illuson 4 NVIDIA Titan X GPUs based on the available trate the general tendency that from top right to botpublic deep-learning framework Pytorch.
tom left, the testing accuracies are constantly rising.
From the perspective of hidden units, we can find that
the capability of distinguishing actions increases with
5 Results
the number of hidden units regardless of how many layers we set. There are some subtle differences in layers
In the following section, we experimentally demon- where no clear linear relations between layers and accustrated the effectiveness of MRRN on action recog- racies. We propose the hypothesis that there may be an
nition benchmarks and compared it with state-of- inverse relation that the property would degrade with
the-art models. This section contains four parts. layers under certain conditions if we make abundant
In the first part, we studied networks with various experiments. Therefore, we could try the setting of 3
hyper-parameters on HMDB-51 dataset. We leveraged layers with 1024 hidden units first for use of SRUs. Due
different-level RRN with two pooling manners to verify to the fact that we imposed the restriction on the input
the contribution to action recognition on both datasets size of SRUs, we did not take the influence of changed
in the second part. The third part is the critical evalu- input vectors into account. Therefore, one thing we
ations of MRRN. Finally, we computed both time com- obliged to think about is the impact of changed inputs
plexity and space complexity to verify the efficiency.
for setting hyper-parameters.
In the Table 1, the SRUs with 3 layers and 1024
hidden
units achieves the best performance by 50.78%.
5.1 Hyper-Parameters
The review of Recurrent Neural Networks [40] reveals
We first investigated the networks with 256, 512 or that the expressive power of hidden states grows ex1024 hidden units and 3, 4 or 5 layers, the mutual ponentially with the number of nodes. Increasing the
combinations between them. In addition, we chose the number of hidden units leads to a great improvement
high-level ResNets as the default extractor so that out- where the performance gap between maximum and
put size of it is 512-dimensional feature vector. That minimum reaches to 6.9%, so we fixed the number of
means the input size of SRUs is fixed as 512 units.
hidden units as 1024 and layers as 3 in the remainder
6

Figure 5: Per-class results on the first split of HMDB-51. This figure shows the per-class results predicted
by different-level RRN. The red, green and blue bars represent the confidence of each class produced by low-,
mid-, high-level RRN.
of this section, although we obtained the bad performance in experiments of low-level and mid-level representations. We will analyze it further in the next
sub-section.

image. Low-level RRN and mid-level RRN concentrate
on redundant details and ignore full object, which leads
to decline. Whereas from the fig.5 we can see that the
performance of high-level RRN is lower than the other
two in particular classes. Consequently, it is beneficial
to integrate different video representations as they are
5.2 Hierarchy Features
highly complementary to each other.
This section aims to verify whether hierarchical feaIn the comparison of time series pooling, mean pooltures can benefit the performance. In addition, we also ing is superior to max pooling in most case. Our analcompared max pooling and mean pooling methods to yses draw the preliminary conclusion that max pooling
understand the differences between them. We picked is sensitive to the noise in the convolutional network beactivation tensors produced by different-level ResNet- cause it takes the maximum from the given dimensions
34 as targets. The results conducted on both datasets of the activation tensor, while mean pooling considers
are listed in Table 2.
entire activation values. So, we assign mean pooling as
default time series pooling, unless otherwise stated.
Table 2: Experiments on Hierarchy Features
Dataset
Pooling Low
Mid
High
5.3 Evaluation of MRRN
mean
26.60 36.80 50.78
HMDB-51
Finally, we make the comparisons between our model
max
25.36 35.82 47.78
and other competitive action recognition models on
mean
46.50 64.16 81.38
both UCF-101 dataset and HMDB-51 dataset. Table
UCF-101
max
45.23 64.96 81.25
3 summarizes the results listed in original papers. We
divide these comparisons into three sets. The upper
set is simple features with linear SVM. The second set
compares models that take only RGB data as input
and the last combines multiple features to predict activities.
It is easy to spot that the deep learning models which
use RGB image merely as input are inferior to the previous hand-crafted features based models though deep
models can capture ample semantic information. One
problem with this is the available training data is so
limited that the deep learning models fail to learn the
intrinsic trait of data. It is worth to mention that
MRRN has not been pre-trained on any video datasets.
In addition to this, our networks are with too many

It is apparent that high-level RRN with mean temporal pooling obtains the supreme performance. In the
case of low-level representations and mid-level representations, we changed the learning rate to 1e-4 for
accelerating the convergence because the convergent
speed of two shallower models with default learning
rate is far slower than the deeper model. The performance gaps between different-level models are so large
that shallower video representations cannot be used to
classify action alone. The study [34] constructs several functions that map the 3D tensor to the 2D tensor
along the channel dimension and discovers that different layers in the network focus on different parts of the
7

Figure 6: Confusion matrix of MRRN model on the first split of UCF-101. Each column represents the
predicted class, and each row represents the ground truth class. The brightness of squares in diagonal represents
the proportion of correct predictions. This figure shows results that the combined model makes predictions on
the first split of UCF-101 and achieves 81.9% accuracy.
hyper-parameters that have the huge impact on the
network performance and we have no clue to search for
the best hyper-parameters but try. That is, we would
be highly competitive with hand-crafted features based
results if we give full play to the potential of MRRN.
In contrast to the model in the second part, the
MRRN boosts the performance to 51.3% and 81.9% on
both datasets, which outperforms the majority of models and achieves the comparable result to the best accuracy. This indicates the combined model can merge
information in different depth and there are some benefits to improve the precision, although the performance
of two shallow models is relatively poor. We observe
that our model outperforms 10.8% and 8.9% better
than Spatial ConvNet [2] on HMDB-51 dataset and
UCF-101 dataset respectively which just learns semantic representations from image appearance, suggesting
additional motion information is beneficial to action
recognition and MRRN can well capture sufficient temporal dynamics in the video sequence. Compared with
cLSTM, MRRN improves from 44.1% to 51.3% and
75.8% to 81.9% on both datasets, which shows MRRN
learns more powerful spatiotemporal representations
than cLSTM. 3D Convolutional network extracts spatiotemporal features from multiple adjacent frames and

Table 3: Comparisons with State-of-the-art
Model
HMDB-51 UCF-101
iDT+SVM [11, 41]
52.1
76.2
iDT+HD encodings [42]
61.1
87.9
Slow Fusion Network [1]
65.4
LRCN [19]
71.1
Spatial ConvNet [2]
40.5
73.0
Soft Attention [26]
41.3
cLSTM [22]
44.1
75.8
High RRN(Our Model)
50.8
81.4
MRRN(Our Model)
51.3
81.9
C3D Model [3]
82.3
Temporal ConvNet [2]
83.7
scLSTM [43]
55.1
84.0
cLSTM [22]
84.3
Two-Stream [2]
59.4
88.0
Feature stacking [44]
65.4
89.1
TDD+iDT [18]
65.9
91.5
RNN-FV+iDT [45]
67.7
94.1

8

trains a linear SVM classifier, which is a little better
than our model. It can be explained by the fact that
multi-class linear SVM classifier has a stronger ability of the classification than softmax classifier in some
case.
Meanwhile, we also make the comparisons with models which use multiple features as input. When combined appearance information with optical flow information, the performance can be enhanced by a large
margin. It is noted that the performance of cLSTM is
lower than our model by 5.6% when only using RGB
data, but superior to ours by 2.4% on UCF-101 dataset
when combined with optical flow images. Despite all
that, these methods have an obvious disadvantage of
computation expensively compared to our model.

Figure 7:

Complexity. Our three sub-models are
trained and tested in parallel, so we compute the complexity independently. The same process is applied to scLSTM
and cLSTM. When computed complexities, we did not take
bias into account for brevity. All experiments followed original settings.

6

The confusion matrix for UCF-101 classification is
shown fig.6 for displaying experimental results intuitively. Each column represents the predicted class,
and each row represents the ground truth class. The
higher brightness of squares in diagonal indicates the
better prediction our model makes and vice versa. This
figure shows the results that MRRN makes predictions
on the first split of UCF-101 and achieves 81.9% accuracy.

5.4

Conclusion

This work proposed a novel model, Multi-Level Recurrent Residual Networks(MRRN), which learns effective video representations for action recognition. We
performed extensive evaluations on the model with various hyper-parameter settings, empirically illustrating
the general tendency of performance. In addition, the
combination of different-level models was shown that
complementary representations further boost the accuracy. Moreover, our model has lower space complexity
and time complexity by employing identity shortcut
compared to the state-of-the-art.

Complexity Analysis

Finally, we calculated time complexity and space
complexity of several competitive methods for verify- 7
Acknowledgements
ing the efficiency. Time complexity indicates the computation complexity that estimates the time taken for
This work was supported partly by the National Nattraining and inferring, while space complexity indicates ural Science Foundation of China(61772067, 61472030,
the number of parameters which a model needs. Note 61471032), the fundamental research funds for the centhat more training data is needed to train the model tral universities (2017JBZ108).
that has higher space complexity. To make a fair comparison with prior works, all experiments followed the
original setting. Complexities are defined as follows:
References

T ime ∼ O(

D
X

Ml2

·

Kl2

· Cl−1 · Cl )

[1] Andrej Karpathy, George Toderici, Sanketh
Shetty, Thomas Leung, Rahul Sukthankar, and
Fei Fei Li. Large-scale video classification with
convolutional neural networks. In IEEE Conference on Computer Vision and Pattern Recognition, pages 1725–1732, 2014.

(12)

l=1
D
X
Space ∼ O(
Kl2 · Cl−1 · Cl )

(13)

l=1

[2] Karen Simonyan and Andrew Zisserman. Twostream convolutional networks for action recognition in videos. Advances in Neural Information
Processing Systems, 1(4):568–576, 2014.

where D denotes the number of convolutional layers,
Ml2 denotes the size of feature map at l layers, Kl2
denotes the size of kernel at l layers, Cl−1 and Cl denote
the number of channels at l − 1 layers and l layers
respectively.

[3] Du Tran, Lubomir Bourdev, Rob Fergus, Lorenzo
Torresani, and Manohar Paluri. Learning spatiotemporal features with 3d convolutional networks. pages 4489–4497, 2014.

The fig 7 shows our three sub-models are lower than
others in both time complexity and space complexity. C3D and scLSTM dramatically add complexity for
abundant 3×3 convolutional kernels while cLSTM uses
large-size feature maps in former layers for improving
accuracies. This illustrates the trade-off between complexity and accuracies.

[4] Guodong Guo and Alice Lai. A survey on still
image based human action recognition. Pattern
Recognition, 47(10):3343–3361, 2014.
9

[5] Piji Li and Jun Ma. What is happening in a still [18] Limin Wang, Yu Qiao, and Xiaoou Tang. Acpicture? In Pattern Recognition, pages 32–36,
tion recognition with trajectory-pooled deep2012.
convolutional descriptors. In Computer Vision
and Pattern Recognition, pages 4305–4314, 2015.
[6] B Alexe, T Deselaers, and V Ferrari. What is an
object? In Computer Vision and Pattern Recog- [19] Jeff Donahue, Lisa Anne Hendricks, Sergio
Guadarrama, Marcus Rohrbach, Subhashini
nition, pages 73–80, 2010.
Venugopalan, Kate Saenko, and Trevor Dar[7] Li Jia Li and Fei Fei Li. What, where and who?
rell. Long-term recurrent convolutional networks
classifying events by scene and object recognition.
for visual recognition and description. CoRR,
In IEEE International Conference on Computer
abs/1411.4389, 2014.
Vision, pages 1–8, 2007.
[20] Christoph Feichtenhofer, Axel Pinz, and Andrew
[8] Bangpeng Yao and Fei Fei Li. Grouplet: A strucZisserman. Convolutional two-stream network futured image representation for recognizing human
sion for video action recognition. pages 1933–1941,
and object interactions. In Computer Vision and
2016.
Pattern Recognition, pages 9–16, 2010.
[21] Xiaojiang Peng and Cordelia Schmid. Multi[9] Navneet Dalal and Bill Triggs. Histograms of oriregion two-stream r-cnn for action detection. In
ented gradients for human detection. In Computer
European Conference on Computer Vision, pages
Vision and Pattern Recognition, 2005. CVPR
744–759, 2016.
2005. IEEE Computer Society Conference on,
[22] Nitish Srivastava, Elman Mansimov, and Ruslan
pages 886–893, 2005.
Salakhutdinov. Unsupervised learning of video
representations using lstms. In International Con[10] Ivan Laptev and Tony Lindeberg. On space-time
ference on International Conference on Machine
interest points. International Journal of Computer
Learning, pages 843–852, 2015.
Vision, 64(2-3):107–123, 2005.
[11] Heng Wang, Alexander Kläser, Cordelia Schmid, [23] Ronald Poppe. Poppe, r.: A survey on visionbased human action recognition. image and vision
and Cheng-Lin Liu. Action Recognition by Dense
computing 28(6), 976-990. 28(6):976–990, 2010.
Trajectories. In IEEE Conference on Computer
Vision & Pattern Recognition, pages 3169–3176,
[24] C Harris. A combined corner and edge detector.
Colorado Springs, United States, June 2011.
Proc Alvey Vision Conf, 1988(3):147–151, 1988.
[12] Heng Wang and Cordelia Schmid. Action recog[25] Ying Zhao, Huijun Di, Jian Zhang, Yao Lu, Feng
nition with improved trajectories. In IEEE InterLv, and Yufang Li. Region-based mixture modnational Conference on Computer Vision, pages
els for human action recognition in low-resolution
3551–3558, 2014.
videos. Neurocomputing, 2017.
[13] H. Kuehne, H. Jhuang, E. Garrote, T. Poggio,
and T. Serre. HMDB: a large video database
for human motion recognition. In Proceedings of
the International Conference on Computer Vision
(ICCV), 2011.

[26] Shikhar Sharma, Ryan Kiros, and Ruslan
Salakhutdinov. Action recognition using visual attention. CoRR, abs/1511.04119, 2015.

[27] Jia Deng, Wei Dong, Richard Socher, Li Jia Li,
Kai Li, and Fei Fei Li. Imagenet: A large-scale
[14] Marcin Marszalek, Ivan Laptev, and Cordelia
hierarchical image database. In Computer ViSchmid. Actions in context. In IEEE Conference
sion and Pattern Recognition, 2009. CVPR 2009.
on Computer Vision & Pattern Recognition, 2009.
IEEE Conference on, pages 248–255, 2009.
[15] Yann Lecun, Lon Bottou, Yoshua Bengio, and [28] Tao Lei and Yu Zhang. Training rnns as fast as
Patrick Haffner. Gradient-based learning applied
cnns. arXiv preprint arXiv:1709.02755, 2017.
to document recognition. In IEEE, pages 2278–
[29] Christian Szegedy, Wei Liu, Yangqing Jia, Pierre
2324, 1998.
Sermanet, Scott Reed, Dragomir Anguelov, Du[16] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E.
mitru Erhan, Vincent Vanhoucke, and Andrew
Hinton. Imagenet classification with deep convoluRabinovich. Going deeper with convolutions. In
tional neural networks. In International ConferComputer Vision and Pattern Recognition, pages
ence on Neural Information Processing Systems,
1–9, 2015.
pages 1097–1105, 2012.
[30] Olga Russakovsky, Jia Deng, Hao Su, Jonathan
[17] Karen Simonyan and Andrew Zisserman. Very
Krause, Sanjeev Satheesh, Sean Ma, Zhiheng
deep convolutional networks for large-scale image
Huang, Andrej Karpathy, Aditya Khosla, and
recognition. Computer Science, 2014.
Michael Bernstein. Imagenet large scale visual
10

recognition challenge. International Journal of [44] Zhenzhong Lan, Ming Lin, Xuanchong Li, AlexanComputer Vision, 115(3):211–252, 2014.
der G. Hauptmann, and Bhiksha Raj. Beyond
gaussian pyramid: Multi-skip feature stacking for
[31] Xavier Glorot and Yoshua Bengio. Understanding
action recognition. pages 204–212, 2014.
the difficulty of training deep feedforward neural
networks. Journal of Machine Learning Research, [45] Guy Lev, Gil Sadeh, Benjamin Klein, and Lior
Wolf. Rnn fisher vectors for action recognition
9:249–256, 2010.
and image annotation. Computer Science, 2015.
[32] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and
Jian Sun. Deep residual learning for image recognition. pages 770–778, 2015.
[33] Gao Huang, Zhuang Liu, Kilian Q Weinberger,
and Van Der Maaten Laurens. Densely connected
convolutional networks. 2016.
[34] Sergey Zagoruyko and Nikos Komodakis. Paying
more attention to attention: Improving the performance of convolutional neural networks via attention transfer. 2016.
[35] Xavier Glorot, Antoine Bordes, Yoshua Bengio,
Xavier Glorot, Antoine Bordes, and Yoshua Bengio. Deep sparse rectifier neural networks. In
International Conference on Artificial Intelligence
and Statistics, pages 315–323, 2012.
[36] Alex Graves. Long short-term memory. Neural
Computation, 9(8):1735, 1997.
[37] Khurram Soomro, Amir Roshan Zamir, and
Mubarak Shah. Ucf101: A dataset of 101 human
actions classes from videos in the wild. Computer
Science, 2012.
[38] Andrew M. Saxe, James L. Mcclelland, and Surya
Ganguli. Exact solutions to the nonlinear dynamics of learning in deep linear neural networks.
Computer Science, 2013.
[39] Diederik P Kingma and Jimmy Ba. Adam: A
method for stochastic optimization. Computer
Science, 2014.
[40] Zachary C. Lipton, John Berkowitz, and Charles
Elkan. A critical review of recurrent neural networks for sequence learning. Computer Science,
2015.
[41] INRIA, Heng Wang, and Cordelia Schmid. Learinria submission for the thumos workshop. 2013.
[42] Xiaojiang Peng, Limin Wang, Xingxing Wang,
and Yu Qiao. Bag of visual words and fusion methods for action recognition: Comprehensive study
and good practice. CoRR, abs/1405.4506, 2014.
[43] Yue Hei Ng, Matthew Hausknecht, Sudheendra
Vijayanarasimhan, Oriol Vinyals, Rajat Monga,
and George Toderici. Beyond short snippets: Deep
networks for video classification. 16(4):4694–4702,
2015.
11

