ActionFlowNet: Learning Motion Representation for Action Recognition
Joe Yue-Hei Ng1
Jonghyun Choi2
Jan Neumann3
Larry S. Davis1
1
2
3
University of Maryland, College Park
Allen Institute for AI
Comcast Labs, DC

arXiv:1612.03052v3 [] 16 Feb 2018

{yhng,lsd}@umiacs.umd.edu

jonghyunc@allenai.org

Abstract
We present a data-efficient representation learning approach to learn video representation with small amount of
labeled data. We propose a multitask learning model ActionFlowNet to train a single stream network directly from
raw pixels to jointly estimate optical flow while recognizing
actions with convolutional neural networks, capturing both
appearance and motion in a single model. Our model effectively learns video representation from motion information
on unlabeled videos. Our model significantly improves action recognition accuracy by a large margin (23.6%) compared to state-of-the-art CNN-based unsupervised representation learning methods trained without external large
scale data and additional optical flow input. Without pretraining on large external labeled datasets, our model, by
well exploiting the motion information, achieves competitive recognition accuracy to the models trained with large
labeled datasets such as ImageNet and Sport-1M.

1. Introduction
Convolutional Neural Networks have demonstrated great
success to multiple visual recognition tasks. With the help
of large amount of annotated data like ImageNet, the network learns multiple layers of complex visual features directly from raw pixels in an end-to-end manner without relying on hand-crafted features. Unlike image labeling, manual video annotation often involves frame-by-frame inspection and temporal trimming of videos that are expensive and
time consuming. This prohibits the technique to be applied
to other problem domains like medical imaging where data
collection is difficult.
We focus on effectively learning video motion representation for action recognition without large amount of external annotated video data. Following previous work [17,
28, 6] that leverages spatio-temporal structure in videos for
unsupervised or self-supervised representation learning, we
are interested in learning video representation from motion
information encoded in videos in addition to semantic la-

bels.

jan neumann@cable.comcast.com

CxTxHxW
3x16x224x224

128x16x56x56
512x4x14x14

…

64x16x112x112

Optical Flow

256x8x28x28

64x16x56x56
128x8x28x28
256x4x14x14
512x2x7x7

…
concat

3D-conv

Action Label

3D-resnet

fc

Figure 1: ActionFlowNet for jointly estaimting optical flow
and recognizing actions. Orange and blue blocks represent
ResNet modules, where blue blocks represents strided convolution. Channel dimension is not shown in the figure.
Learning motion representation on videos from raw pixels is challenging. With large scale datasets such as Sports1M [10] and Kinetics [11], one could train a high capacity classifier to learn complex motion signatures for action recognition by extending image based CNN architectures with 3D convolutions for video action recognition [10, 26, 2]. However, while classification loss is an excellent generic appearance learner for image classification,
it is not necessarily the most effective supervision for learning motion features for action recognition. As shown in [2],
even with large amount of labeled video data, the model still
benefits from additional optical flow input stream. This suggests that the model is ineffective in learning motion representation for action recognition from video frames, and thus
alternative approach should be explored for learning video
representation.
Two-stream convolutional neural networks, which separately learn appearance and motion by two convolutional
networks on static images and optical flow respectively,
show impressive results on action recognition [22]. The
separation, however, fails to learn the interaction between
the motion and the appearance of objects, and introduces
additional complexity of computing the flow to the classification pipeline. In addition, human visual system does
not take optical flow as front end input signals but infer the
motion from raw intensities internally. Therefore, we focus
to learn both motion features and appearance directly from

raw pixels without hand-crafted flow input.
Encouraged by the success on estimating optical flow
with convolutional neural networks [7], we train a single stream feed-forward convolutional neural network ActionFlowNet - for jointly recognizing actions and estimating optical flow. Specifically, we formulate the learning
problem as multitask learning, which enables the network to
learn both appearance and motion in a single network from
raw pixels. The proposed architecture is illustrated in Figure 1. With the auxiliary task of optical flow learning, the
network effectively learns useful representations from motion modeling without a large amount of human annotation.
Based on the already learned motion modeling, the model
then only requires action annotations as supervision to learn
action class specific details, which results in requiring less
annotation to perform well for action recognition.
Our experiments and analyses show that our model successfully learns motion features for action recognition and
provide insights on how the learned optical flow quality affects action classification. We demonstrate the effectiveness
of our learned motion representation on two standard action
recognition benchmarks - UCF101 and HMDB51. Without
providing external training data or fine-tuning from already
well-trained models with millions of samples, we show
that jointly learning action and optical flow significantly
boosts action recognition accuracy compared to state-ofthe-art representation learning methods trained without external labeled data. Remarkably, our model outperforms
the models trained with large datasets Sports-1M pretrained
C3D by 1.6% on UCF101 dataset, showing the importance
of feature learning algorithms.

2. Related Work
Over the past few years, action recognition accuracy
has been greatly improved by learned features and various
learning models utilizing deep networks. Two-stream network architecture was proposed to recognize action using
both appearance and motions separately [22]. A number
of follow up methods have been proposed based on twostream networks that further improved action recognition
accuracies [5, 31, 30, 4, 18]. Our work is motivated by their
success in incorporating optical flow for action recognition,
but we focus on learning from raw pixels instead of relying
on hand-crafted representations.
Optical flow encodes motion between frames and is
highly related to action recognition. Our model is motivated by the success of FlowNet [7] and 3D convolutions
for optical flow estimation in videos [27], but emphasizes
on improving action recognition.
Pre-training the network with a large dataset helps to
learn appearance signatures for action recognition. Karpathy et al. proposed a “Slow Fusion” network for large scale
video classification [10]. Tran et al. trained a 3D convo-

lutional neural network (C3D) with a large amount of data
and showed the learned features are generic for different
tasks [26]. Recently, Carreira and Zisserman trained I3D
models [2] on the Kinetics dataset [11] and achieved strong
action recognition performance. In contrast, since training
networks on such large scale datasets is extremely computationally expensive, we focus on learning from small
amounts of labeled data. With only small amount of labeled data, we show that our model performs competitive
to models trained with large datasets.
Leveraging videos as a source for unsupervised learning has been suggested to learn video representations without large labeled data. Different surrogate tasks have been
proposed to learn visual representations from videos without any labels. Wang et al. trained a network to learn visual similarity for patches obtained from visual tracking in
videos [32]. Misra et al. trained a network to differentiate
the temporal order of different frames from a video [17].
Jacob et al. learned apperance features by predicting the future trajectories in videos [29]. Fernando et al. proposed
Odd-One-Out networks (O3N) to identify video sequences
that are out of order for self-supervised learning [6]. Our
work, similarly, uses video as an additional source for learning visual representation. However, in contrast to previous
work which focused on learning visual representations for
a single image, we learn motion representations for videos
which models more than a single frame. Vondrick et al.
used a Generatie Adversarial Network to learn a generative
model for video [28]. We focus on learning motion representations but not video generation.
Independent to our work, Diba et al. trained a two stream
network with flow estimation [3]. They based their network
on C3D with a two-stream architecture. Our work employs
a single stream network to learn both appearance and motion. While we both estimate motion and recognize actions
in the same model, we focus on learning motion representations without pretraining on large labeled datasets and provide more analysis to learn flow representations for action
recognition.

3. Approach
We propose a single end-to-end model to learn both motions and action classes simultaneously. Our primary goal
is to improve action classification accuracy with the help of
motion information; we use optical flow as a motion signature. Unlike previous methods that utilize externally computed optical flow as the input to their models, we only use
the video frames for input and simultaneously learn the flow
and class labels.

3.1. Multi-frame Optical Flow with 3D-ResNet
Fischer et al. proposed FlowNet [7] that is based on convolutional neural networks to estimate high quality optical

flow. Tran et al. proposed to use 3D convolution and deconvolution layers to learn multi-frame optical flow from
videos [27]. In addition, He et al. introduced residual networks (ResNet) to train a deeper convolutional neural network model by adding shortcut connections [8].
In addition to the benefit of easy training, ResNet is fully
convolutional, so is easily applied to pixel-wise prediction
of optical flow, unlike many architectures with fully connected layers including AlexNet [13] and VGG-16 [23]. In
contrast to other classification architectures like AlexNet
and VGG-16, which contains multiple max pooling layers
that may harm optical flow estimation, the ResNet architecture only contains one pooling layer right after conv1. We
believe the reduced number of pooling layers makes ResNet
more suitable for optical flow estimation where spatial details need to be preserved. Specifically, we use an 18 layers
ResNet, which is computationally efficient with good classification performance [8].
Taking advantage of ResNet for flow estimation, we extend ResNet-18 to 3D-ResNet-18 for multi-frame optical
flow estimation by replacing all k × k 2D convolutional
kernels with extra temporal dimension k × k × 3, inspired
by [27]. The deconvolution layers in the decoder are extended similarly. Skip connections from encoder to decoder
are retained as in [7] to obtain higher resolution information in the decoder. Unlike [7], we only use the loss on
the highest resolution to avoid downsampling in the temporal dimension. We do not apply temporal max pooling
suggested in [26, 27], but use only strided convolutions to
preserve temporal details. After the third residual block, the
temporal resolution is reduced by half when the spatial resolution is reduced.
Future Prediction. In addition to computing the optical flow between the T input frames, we train the model to
predict the optical flow on the last frame, which is the optical flow between the T th and (T + 1)st frames. There are
two benefits of training the model to predict the optical flow
of the last frame: 1) It is practically easier to implement a
model with the same input and output sizes, since the output sizes of deconvolution layers are usually multiples of
the inputs; and 2) Semantic reasoning is required for the
model to extrapolate the future optical flow given the previous frames. This possibly trains the model to learn better
motion features for action recognition, as also suggested by
previous work [29], which learned appearance feature by
predicting the future.
Following [7], the network is optimized over the endpoint error (EPE), which is the sum of L2 distance between
the ground truth optical flow and the obtained flow over all
pixels. The total loss for the multiple frame optical flow
model is the EPE of T output optical flow frames:
T X
X
koj,t,p − o[
(1)
j,t,p k2 ,
t=1

p

where oj,t,p is 2-dimensional optical flow vector of the tth
and the (t + 1)st frame in the j th video at pixel p.
Note that the T th optical flow frame oj,t is the future
optical flow for the T th and (T + 1)st input frames, where
the (T + 1)st frame is not given to the model.

3.2. ActionFlowNet
Knowledge Transfer by Finetuning. Finetuning a pretrained network is a common practice to transfer knowledge from different datasets and tasks. Unlike previous
work, where knowledge transfer has been accomplished between very similar tasks (image classification and detection or semantic segmentation), knowledge transfer in our
model is challenging since the goals of pixel-wise optical
flow and action classification are not obviously compatible.
We transfer the learned motion by initializing the classification network using a network trained for optical flow estimation. Since the network was trained to predict optical
flow, it should encode motion information in intermediate
levels which support action classification. However, finetuning a pretrained network is known to have the problem of
catastrophic forgetting. Specifically, when training the network for action recognition, the originally initialized flow
information could be destroyed when the network adapts
the appearance information. We prevent catastrophic forgetting by using the multitask learning framework.
ActionFlowNet. To force the model to learn motion features while training for action recognition, we propose
a multitask model ActionFlowNet, which simultaneously
learns to estimate optical flow, together with predicting the
future optical flow of the last frame, and action classification to avoid catastrophic forgetting. With optical flow as
supervision, the model can effectively learn motion features
while not relying on explicit optical flow computation.
In our implementation, we take 16 consecutive frames
as input to our model. In the last layer of the encoder,
global average pooling across the spatial-temporal feature
map, with size 512 × 2 × 7 × 7, is employed to obtain a
single 512 dimensional feature vector, followed by a linear
softmax classifier for action recognition. The architecture
is illustrated in Figure 1. The multitask loss is given as follows:
MT-Lossj = −1(yj = ybj ) log p(ybj ) +
{z
}
|
Classification Loss

λ

T X
X
t=1

|

koj,t,p − o[
j,t,p k2 , (2)

p

{z

Flow Loss

}

where 1(·) is a indicator function, yj and ybj are the
groundtruth and predicted action labels respectively of the
j th video. λ is a hyper-parameter balancing the classification loss and the flow loss, where optical flow estimation

can be seen as a regularizer for the model to learn motion
feature for classification.
Although previous work on multitask learning [16] suggests that sharing parameters of two different tasks may hurt
performance, this architecture performs well since optical
flow is known empirically to improve video action recognition significantly. In addition, our architecture contains
multiple skip connections from lower convolutional layers
to decoder. This allows higher layers in the encoder to focus on learning more abstract and high level features, without constraining them to remembering all spatial details for
predicting optical flow, which is beneficial for action recognition. This idea is central to Ladder Networks [20] which
introduced lateral connections to learn denoising functions
and significantly improved classification performance.
It is worth noting that this is a very general architecture
and requires minimal architectural engineering. Thus, it can
be trivially extended to learn more tasks jointly to adapt
knowledge from different domains.
ActionFlowNet Inference. During inference for action
classification, optical flow estimation is not required since
the motion information is already learned in the encoder.
Therefore, the decoder can be removed and only the forward pass of the encoder and the classifier are computed.
If the same backbone architecture is used, our model runs
at the same speed as a single-stream RGB network without extra computational overhead. Since the optical flow
estimation and flow-stream CNN are not needed, it is more
efficient than two-stream counterparts.

3.3. Two-Frame Based Models
In this section, we propose various models that take two
consecutive input frames. Experimenting with two-frame
models has three benefits. First, when there are multiple frames in the input, it is difficult to determine whether
the performance improvement comes from motion modeling or aggregating long term appearance information. Thus
for better analysis, it is desirable to use the two frame input. Second, training two-frame models is computationally
much more efficient than multi-frame models which take N
video frames and output N − 1 optical flow images. Third,
we can measure the effectiveness of external large scale optical flow datasets, such as the FlyingChairs dataset [7],
which provide ground-truth flow on only two consecutive
frames, for action recognition.
Learning Optical Flow with ResNet. Similarly, we use
ResNet-18 as our backbone architecture and learn optical
flow. Like FlowNet-S [7], we concatenate two consecutive
frames to produce a 6(ch) × 224(w) × 224(h) input for our
two frames model. At the decoder, there are four outputs
with different resolutions. The total optical flow loss is the
weighted sum of end-point error at multiple resolutions per
the following equation:

4
X

αr

X

r=1

[
(r)
(r)
koj,t,p − oj,t,p k2 ,

(3)

p

(r)

where oj,t,p is the optical flow vector of the rth layer output
and αr is the weighting coefficient of the rth optical flow
output. We refer to this pre-trained optical flow estimation
network as FlowNet.
We first propose an architecture to classify actions on
top of the optical flow estimation network, which we call
the Stacked Model. Then, we present the two-frame version
of ActionFlowNet to classify the actions and estimate the
optical flow, which we call the ActionFlowNet-2F.
3.3.1

Stacked Model

A straightforward way to use the trained parameters from
FlowNet is to take the output of FlowNet and learn a CNN
on top of the output, as shown in Figure 2. This is reminiscence of the temporal stream in [22] which learns a CNN
on precomputed optical flow. If the learned optical flow has
high quality, it should give similar performance to learning
a network on optical flow.
6x224x224
64x56x56

128x28x28
256x14x14
512x7x7

Optical
Flow 64x56x56

128x28x28
256x14x14

Action
Labels

512x7x7

FC

Softmax

ResNet

Figure 2: Network structure of the ‘Stacked Model’.
Since the output of FlowNet has 4 times lower resolution
than the original image, we remove the first two layers of
the CNN (conv1 and pool1) and stack the network on top
of it. We also tried to upsample the flow to the original
resolution and use the original architecture including conv1
and pool1, but this produces slightly worse results and is
computationally more expensive.
The stacked model introduces about 2x number of parameters compared to the original ResNet, and is also 2x
more expensive for inference. It learns motion features by
explicitly including optical flow as an intermediate representation, but cannot model appearance and motion simultaneously, similar to learning a CNN on precomputed optical
flow.
3.3.2

ActionFlowNet-2F

The multitask ActionFlowNet-2F architecture, as illustrated
in Figure 3, is based on the two-frame FlowNet with additional classifier. Similar to ActionFlowNet, classification is
performed by average pooling the last convolutional layer
in the encoder followed by a linear classsifier.

56x56x64
128x28x28

6x224x224
64x56x56

Optical
Flow

256x14x14
128x28x28
256x14x14
512x7x7 R2

R3

R4

R1

Action
Label

ResNet
FC

Softmax

Figure 3: Network structure of the ActionFlowNet-2F
Just as with the stacked model, the loss function is defined for each frame. For the tth frame in the j th video the
loss is defined as a weighted sum of classification loss and
optical flow loss:
MT-Lossj,t = −1(yj = ybj ) log p(ybj ) +
|
{z
}
Classification Loss

λ

4
X
r=1

|

4. Experiments

αr

X

[
(r)
(r)
koj,t,p − oj,t,p k2 , (4)

p

{z

Flow Loss

}

4.1. Datasets
We use two publicly available datasets, UCF101 and
HMDB51, to evaluate action classification accuracy. The
UCF101 dataset contains 13,320 videos with 101 action
classes [24]. The HMDB51 contains 6,766 videos with 51
action categories [14]. As the number of training videos
in HMDB51 is small, we initialized our models trained on
UCF101 and fine-tuned for HMDB51 similar to [22]
The UCF101 and HMDB51 do not have groundtruth optical flow annotation. Similar to [27], we use EpicFlow [21]
as a psuedo-groundtruth optical flow to train the motion part
of the network.
To experiment models with better learned the motion
signature, we also use FlyingChairs dataset [7] as it has
groundtruth optical flow since it is a synthetic dataset.
The FlyingChairs dataset contains 22,872 image pairs and
ground truth flow from synthetically generated chairs on
real images. We use the Sintel dataset [1], which provides
dense groundtruth optical flow, to validate the quality of optical flow models.

4.2. Experimental Setup
Overfitting Prevention. We use different data augmentations on different datasets and tasks. On the FlyingChairs
dataset for optical flow estimation, we augment the data
using multi-scale cropping, horizontal flipping, translation
and rotation following [7]. On the UCF101 dataset for optical flow estimation, we use multi-scale cropping and horizontal flipping, but do not use translation and rotation in
order to maintain the original optical flow distribution in
the data. On UCF101 dataset for action recognition, we use

color jittering [25], multi-scale cropping and horizontal flipping. Dropout is applied to the output of the average pooling
layer before the linear classifier with probability 0.5.
Optimization and Evaluation. The models are trained using Adam [12] for 40,000 iterations with batch size 128 and
learning rate 1 × 10−4 . For evaluation, we sample 25 random video segments from a video and run a forward pass to
the network on the 10-crops (4 corners + center with their
horizontal reflections) and average the prediction scores.

4.3. Improving Action Recognition
We first evaluate the action recognition accuracy by
the various proposed two-frame models described in Section 3.3, and then the multi-frame models in Section 3.2,
on both UCF101 and HMDB51 datasets. All models take
RGB inputs only without external optical flow inputs. The
recognition accuracies are summarized in Table 1.
Method

UCF101

HMDB51

Two-frame Models
Scratch
FlowNet fine-tune
Stacked
ActionFlowNet-2F (UCF101)
ActionFlowNet-2F (FlCh+UCF101)
ImageNet pretrained ResNet-18

51.3
66.0
69.6
70.0
71.0
80.7

23.9
29.1
42.4
42.4
42.6
47.1

Multi-frame Models
Multi-frame FlowNet fine-tune
ActionFlowNet (UCF101)
Sports-1M pretrained C3D [26]
Kinetics pretrained I3D [2]

80.8
83.9
82.3
95.6

50.6
56.4
53.5
74.8

Table 1: Action recognition accuracies of our models on
UCF101 and HMDB51 datasets (split 1). FlCh denotes
FlyingChairs dataset. “ActionFlowNet-2F (UCF101)” denotes its FlowNet part is pretrained on UCF101, and
“ActionFlowNet-2F (FlCh+UCF101)” denotes its FlowNet
part is pretrained on FlyingChairs dataset. All ActionFlowNets are then learned on UCF101 dataset for action
and flow. For reference, we additionally show the results
trained with large scale datasets [27, 2], but it is not directly
comparable since our models are trained with significantly
less annotation.
Two-frame Models. ‘Scratch’ is a ResNet-18 model
that is trained from scratch (random initialization) using UCF101 without any extra supervision, which represents the baseline performance without motion modeling. ‘FlowNet fine-tune’ is a model that is pretrained from
UCF101 for optical flow only, and then fine-tuned with action classification, which captures motion information by
initialized FlowNet. ‘Stacked’ is a stacked classification
model on top of optical flow output depicted in Figure 2.
Its underlying FlowNet is trained with UCF101 and is fixed

to predict optical flow, so only the CNN classifier on top
is learned. ‘ActionFlowNet-2F’ is the multitask model depicted in Figure 3, which is trained for action recognition
and optical flow estimation to learn both motion and appearance. We trained two versions of ActionFlowNet-2F:
one with FlowNet pretrained on UCF101 and one on FlyingChairs dataset.
As shown in the table, all proposed models - ‘FlowNet
fine-tune’, ‘Stacked’ and ‘ActionFlowNet-2F’ significantly
outperform ‘Scratch’ . This implies that our models can
take advantage of the learned motion for action recognition,
which is difficult to learn implicitly from action labels.
Both the Stacked model and two ActionFlowNet-2Fs
outperform the finetuning models by a large margin (up to
5.0% in UCF101 and up to 13.5% in HMDB51). As all
models are pretrained from the high quality optical flow
model, the results show that the knowledge learned from
previous task is prone to be forgotten when learning new
task without multitask learning. With extra supervision
from optical flow estimation, multitask models regularize
the action recognition with the effort of learning the motion
features.
While the Stacked model performs similarly to
ActionFlowNet-2F when trained only on UCF101,
ActionFlowNet-2F is much more compact than the Stacked
model, containing only approximately half the number of
parameters of the Stacked model. When ActionFlowNet2F is first pretrained with FlyingChairs, which predicts
better quality optical flow in EPE, and finetuned with the
UCF101 dataset, it further improves accuracy by 1%. This
implies that our multitask model is capable of transferring
general motion information from other datasets to improve
recognition accuracy further.
Our ActionFlowNet-2F still performs inferior compared to ResNet pretrained on ImageNet, especially in
UCF101 (71.0% vs 80.7%) because of the rich background
context appearance in the dataset. When evaluated on
HMDB51, where the backgrounds are less discriminative,
our ActionFlowNet-2F is only slightly behind the ImageNet
pretrained model (42.6% vs 47.1%), indicating that our
model learns strong motion features for action recognition.
Multi-frame Models. We train 16-frame ActionFlowNet on UCF101. The results are shown in the lower
part of Table 1. By taking more frames per model, our
multi-frame models significantly improve two-frame models (83.9% vs 70.0%). This confirms previous work [10, 19]
that taking more input frames in the model is important.
Remarkably, without pretraining on large amounts of labeled data, our ActionFlowNet outperforms the ImageNet
pretrained single frame model and Sports-1M pretrained
C3D. Our ActionFlowNet gives 1.6% and 2.9% improvements over C3D on UCF101 and HMDB51 repsectively.
The recently published I3D models [2] achieved strong

performance by training on the newly released Kinetics
dataset [11] with large amount of clean and trimmed labeled video data and performing 3D convolutions on 64 input frames instead of 16 frames. Although the I3D model
achieved better results compared to previous work, their
RGB model could still benefit from optical flow inputs,
which indicates that even with large amount of labeled data
the I3D model does not learn motion features effectively.
It should be noted that there is prior work that gives better results with the use of large scale datasets like ImageNet
and Kinetics dataset [2], or with the help of external optical
flow input [22]. Those results are not directly comparable
to us because we are using a significantly smaller amount
of labeled data - only UCF101 and HMDB51. Nevertheless, our method shows promising results for learning motion representations from videos. Even with only a small
amount of labeled data, our action recognition network outperforms methods trained with a large amount of labeled
data with the exception of the recently trained I3D models [2] which used ImageNet and Kinetics dataset [11]. We
envision the performance of ActionFlowNet would further
improve when trained on larger datasets like Kinetics and
taking more input frames in the model.
Method
ResNet-18 Scratch
VGG-M-2048 Scratch [22]
Sequential Verification [17]
VGAN [28]
O3N [6]
OPN [15]
FlowNet fine-tuned (ours)
ActionFlowNet-2F (ours)
ActionFlowNet (ours)

UCF101 Accuracy
51.3
52.9
50.9
52.1
60.3
59.8
66.0
70.0
83.9

Table 2: Results on UCF101 (split 1) from single stream
networks with raw pixel input and without pretraining on
large labeled dataset.
Comparison to state-of-the-arts. We compare our approach to previous work that does not perform pretraining
with external large labeled datasets in Table 2 on UCF101.
All models are trained only with UCF101 labels with different unsupervised learning methods. Our models significantly outperform previous work that use videos for unsupervised feature learning [17, 28, 6, 15]. Specifically,
even with only our two-frame fine-tuned model on UCF101,
the model obtain more than 5.9% improvement compared
to Sequential Verification, VGAN and O3N, indicating the
importance of motion in learning video representations.
When combined with multitask learning, the performance
improves to 70.0%. Finally, when extending our model to
16 frames by 3D convolutions, the performance of ActionFlowNet further boost to 83.9%, giving a 23.6% improve-

(a) Image

(b) Output flow from (c) ImageNet Model: (d) ActionFlowNet-2F:
ActionFlowNet-2F
Appearance Only
Motion and Appearance

Figure 4: Visualization of important regions for action recognition. Our ActionFlowNet-2F discovers the regions where the
motions are happening to be important while ‘Appearance Only’ captures discriminative regions based on the appearance.
tion, while the ImageNet pretrained network relies more on
background appearance, which may not directly relate to
the action itself. However, when appearance is discriminative - for example the writing on the board in the last example - our model can also focus on appearance, which is not
possible for models that learn from optical flow only.
4.3.2

4.3.1 Learning Motions for Discriminative Regions
We visualize what is learned from the multitask network by
using the method from [33] by using a black square to occlude the frames at different spatial locations and compute
the relative difference between classification confidence before and after occlusion. We visualize the two-frame based
ActionFlowNet-2F for more straightforward visualization.
We compare the discriminative regions discovered by
our multitask network with ones by the ImageNet pretrained
ResNet-18, which only models the discriminative appearances without motion. Figure 4 shows example results. The
visualization reveals that our model focuses more on mo-

0.5
0.0
WallPushups
ApplyEyeMakeup
BodyWeightSquats
ShavingBeard
PlayingDaf
Haircut
FrontCrawl
PlayingTabla
BlowDryHair
Knitting
YoYo
PlayingDhol
Shotput
ThrowDiscus
HandstandPushups
JavelinThrow
LongJump
PoleVault
Lunges
HeadMassage
SalsaSpin
JumpingJack
BreastStroke
TennisSwing
MilitaryParade
CleanAndJerk
BoxingSpeedBag
PlayingFlute
CricketShot
PullUps
Hammering
CuttingInKitchen
HighJump
BenchPress
PizzaTossing
Drumming
Mixing
Diving
Typing
PlayingViolin
HulaHoop
HorseRace
PommelHorse
VolleyballSpiking
Billiards
RockClimbingIndoor
RopeClimbing
Biking
BasketballDunk
BandMarching
Skijet
TrampolineJumping
HorseRiding
IceDancing
SoccerPenalty
JugglingBalls
TableTennisShot
GolfSwing
ParallelBars
CricketBowling
FloorGymnastics
Rowing
BrushingTeeth
HandstandWalking
SumoWrestling
Surfing
BlowingCandles
ApplyLipstick
SkyDiving
PushUps
PlayingPiano
Rafting
WritingOnBoard
Swing
SoccerJuggling
Punch
BalanceBeam
HammerThrow
PlayingGuitar
Bowling
UnevenBars
Kayaking
PlayingSitar
Archery
Skiing
FrisbeeCatch
PlayingCello
BabyCrawling
FieldHockeyPenalty
StillRings
CliffDiving
WalkingWithDog
Fencing
TaiChi
BoxingPunchingBag
JumpRope
SkateBoarding
Nunchucks
BaseballPitch
Basketball
MoppingFloor

ment over the best previous work. This shows that explicitly
learning motion information is important for learning video
representations.

Figure 5 shows the optical flow estimation and prediction
results from our multi-frame model. Although the model
does not have accurate optical flow groundtruth for training, the optical flow quality is fairly good. The model predicts reasonable future optical flow, which shows semantic
understanding from the model to the frames in addition to
simply performing matching between input frames.

(a) ActionFlowNet vs C3D

0.5
0.0
SoccerJuggling
JumpRope
CricketShot
PullUps
JumpingJack
BodyWeightSquats
Lunges
Nunchucks
WallPushups
FrontCrawl
PommelHorse
FloorGymnastics
TrampolineJumping
SalsaSpin
Haircut
CleanAndJerk
Archery
BoxingSpeedBag
BenchPress
HeadMassage
Swing
HandstandWalking
Drumming
PushUps
PoleVault
Knitting
HighJump
GolfSwing
RockClimbingIndoor
Shotput
JavelinThrow
RopeClimbing
YoYo
CricketBowling
HandstandPushups
Skijet
SkyDiving
Surfing
HulaHoop
BabyCrawling
VolleyballSpiking
WalkingWithDog
Kayaking
ThrowDiscus
BlowDryHair
Biking
LongJump
JugglingBalls
ApplyEyeMakeup
Typing
BreastStroke
SumoWrestling
TableTennisShot
Billiards
Diving
BasketballDunk
Basketball
Punch
PlayingTabla
BandMarching
HammerThrow
HorseRace
ParallelBars
TennisSwing
HorseRiding
IceDancing
Bowling
SoccerPenalty
PizzaTossing
MilitaryParade
UnevenBars
PlayingViolin
PlayingPiano
WritingOnBoard
PlayingDaf
FieldHockeyPenalty
Skiing
FrisbeeCatch
Rowing
Hammering
PlayingDhol
PlayingFlute
ApplyLipstick
StillRings
ShavingBeard
Mixing
CuttingInKitchen
BalanceBeam
TaiChi
PlayingGuitar
Fencing
PlayingSitar
Rafting
CliffDiving
SkateBoarding
PlayingCello
BoxingPunchingBag
BaseballPitch
BlowingCandles
MoppingFloor
BrushingTeeth

Figure 5: Optical flow and future prediction outputs from
our multi-frame model. The 1st and 3rd row shows an
example of input videos, and the 2nd and 4th row shows
the corresponding optical flow outputs. The last optical
flow output frames (in red border) are extrapolated rather
than computed within input frames. Only last 8 frames are
shown per sample due to space limit.

Optical Flow and Future Prediction

(b) ActionFlowNet vs ImageNet pretrained ResNet-18

Figure 6: Classwise accuracy improvement by ActionFlowNet over pretrained models. The blue bars show positive improvements and the red ones show otherwise.

4.3.3 Classes Improved By Learning Motions
We compare the per class accuracy for ActionFlowNet, ImageNet pretrained model and C3D. Not all action classes are
motion-centric - objects and their contextual (background)
appearances provide more discriminative information for
some classes [9], which can greatly benefit from large
amounts of labeled data. As shown in Figure 6, our model
better recognizes action classes with simple and discriminative motion like WallPushups and ApplyEyeMakeup, while
C3D and ImageNet models perform better on classes with
complex appearance like MoppingFloor and BaseballPitch.

(a) Frame with movements
highlighted with red

(b) EpicFlow

(c) FlowNet on FlyingChairs

(d) FlowNet on UCF101

4.4. Recognition and Optical Flow Quality
In this section, we study the effects of different optical flow models for action recognition based on the twoframe models. We train our optical flow models on FlyingChairs or UCF101 and evaluate their accuracies on the
Sintel dataset (similar to [7] that trains the model on FlyingChairs but tests on other datasets).
We investigate how the quality of the learned optical flow
affects action recognition. Since optical flow in the multitask model is collaboratively learned with the recognition
task, the quality of optical flow in the multitask model does
not directly affect recognition accuracy. Thus, we use our
Stacked model learned with different datasets, fix the optical flow part and train the classification part in the network
shown in Figure 2. We compare the end-point-error of different optical flow learners and the corresponding classification accuracy in Table 3.
Method
Stacked on FlyingChairs
Stacked on UCF101
ResNet on EpicFlow

EPE on Sintel
9.12
11.84
6.29

Classification
Accuracy (%)
51.7
69.6
77.7

Table 3: Comparison between End-Point-Error (EPE, lower
is better) and the classification accuracy. Interestingly, better optical flow does not always result in better action recognition accuracy. Refer to the text for discussion.
Action Recognition with Learned Flow. Surprisingly,
even with lower end-point-error the Stacked model pretrained on FlyingChairs performs significantly worse than
the one pretrained on UCF101 dataset (51.7% vs 69.6%),
as shown in Table 3. Compared to the model directly taking high quality optical flow as input (77.7%), our models
are still not as good as training directly on optical flow. We
believe this is because the quality of learned optical flow is
not high enough.
To understand how the learned optical flow affects action recognition, we qualitatively observe the optical flow
performance in Figure 7. Even though the end-point error on Sintel of the FlowNet pretrained on FlyingChairs
is low, the estimated optical flow has lots of artifacts in

Figure 7: Qualitative comparison of flow outputs. It shows
an example of small motion, where the maximum magnitude of displacement estimated from EpicFlow is only about
1.6px. FlowNet trained on FlyingChairs dataset fails to estimate small motion, since the FlyingChairs dataset consists
of large displacement flow.
the background and the recognition accuracy on top of that
is correspondingly low. We believe the reason is that the
FlyingChairs dataset mostly consists of large displacement
flow, and therefore the model performs badly on estimating
small optical flow, which contributes less in the EPE metric
when averaged over the whole dataset. This is in contrast to
traditional optimization based optical flow algorithms that
can predict small displacements well but have difficulties
for large displacements.
In addition, traditional optical flow algorithms such as
TV-L1 and EpicFlow explicitly enforce smoothness and
constancy. They are able to preserve object shape information when the flow displacements are small, which is important for action recognition. While our models perform
comparably to traditional optical flow algorithms in terms
of endpoint error, our model is not optimized for preserving
flow smoothness. This shows that end-point-error of optical
flow in public dataset may not be a good indicator of action
classification performance, since shape preservation is not
accounted for in the metric.

5. Conclusion
We presented a multitask framework for learning action
with motion flow, named ActionFlowNet. By using optical
flow as supervision for classification, our model captures
motion information while not requiring explicit optical flow
computation as input. Our model significantly outperforms
previous feature learning methods trained without external
large scale data and additional optical flow input.

Acknowledgments
This research was supported in part by funds provided from
the Office of Naval Research under grant N000141612713
entitled “Visual Common Sense Reasoning for Multi-agent
Activity Prediction and Recognition”.

References
[1] D. J. Butler, J. Wulff, G. B. Stanley, and M. J. Black. A
naturalistic open source movie for optical flow evaluation.
In ECCV, 2012. 5
[2] J. Carreira and A. Zisserman. Quo vadis, action recognition?
a new model and the kinetics dataset. In CVPR, 2017. 1, 2,
5, 6
[3] A. Diba, A. M. Pazandeh, and L. V. Gool. Efficient twostream motion and appearance 3d cnns for video classification. arXiv preprint arXiv:1608.08851, 2016. 2
[4] C. Feichtenhofer, A. Pinz, and R. Wildes. Spatiotemporal
residual networks for video action recognition. In NIPS,
2016. 2
[5] C. Feichtenhofer, A. Pinz, and A. Zisserman. Convolutional
two-stream network fusion for video action recognition. In
CVPR, 2016. 2
[6] B. Fernando, H. Bilen, E. Gavves, and S. Gould. Selfsupervised video representation learning with odd-one-out
networks. In CVPR, 2017. 1, 2, 6
[7] P. Fischer, A. Dosovitskiy, E. Ilg, P. Häusser, C. Hazırbaş,
V. Golkov, P. van der Smagt, D. Cremers, and T. Brox.
FlowNet: Learning Optical Flow with Convolutional Networks. In ICCV, 2015. 2, 3, 4, 5, 8
[8] K. He, X. Zhang, S. Ren, and J. Sun. Deep Residual Learning
for Image Recognition. CVPR, 2016. 3
[9] M. Jain, J. van Gemert, and C. Snoek. What do 15,000 object
categories tell us about classifying and localizing actions? In
CVPR, 2015. 8
[10] A. Karpathy, G. Toderici, S. Shetty, T. Leung, R. Sukthankar,
and L. Fei-Fei. Large-scale Video Classification with Convolutional Neural Networks. In CVPR, 2014. 1, 2, 6
[11] W. Kay, J. Carreira, K. Simonyan, B. Zhang, C. Hillier, S. Vijayanarasimhan, F. Viola, T. Green, T. Back, P. Natsev, et al.
The kinetics human action video dataset. arXiv preprint
arXiv:1705.06950, 2017. 1, 2, 6
[12] D. Kingma and J. Ba. Adam: A Method for Stochastic Optimization. In ICLR, 2015. 5
[13] A. Krizhevsky, I. Sutskever, and G. E. Hinton. ImageNet
Classification with Deep Convolutional Neural Networks. In
NIPS, 2012. 3
[14] H. Kuehne, H. Jhuang, E. Garrote, T. Poggio, and T. Serre.
HMDB: A large video database for human motion recognition. In ICCV, 2011. 5
[15] H.-Y. Lee, J.-B. Huang, M. Singh, and M.-H. Yang. Unsupervised representation learning by sorting sequences. In
ICCV, 2017. 6
[16] I. Misra, A. Shrivastava, A. Gupta, and M. Hebert. Crossstitch Networks for Multi-task Learning. In CVPR, 2016. 4

[17] I. Misra, L. C. Zitnick, and M. Hebert. Shuffle and learn:
unsupervised learning using temporal order verification. In
ECCV, 2016. 1, 2, 6
[18] J. Y.-H. Ng and L. S. Davis. Temporal difference networks
for video action recognition. In WACV, 2018. 2
[19] J. Y.-H. Ng, M. Hausknecht, S. Vijayanarasimhan,
O. Vinyals, R. Monga, and G. Toderici. Beyond Short Snippets: Deep Networks for Video Classification. In CVPR,
2015. 6
[20] A. Rasmus, H. Valpola, M. Honkala, M. Berglund, and
T. Raiko. Semi-Supervised Learning with Ladder Networks.
In NIPS, 2015. 4
[21] E. Revaud, P. Weinzaepfel, Z. Harchaoui, and C. Schmid.
EpicFlow: Edge-Preserving Interpolation of Correspondences for Optical Flow. In CVPR, 2015. 5
[22] K. Simonyan and A. Zisserman. Two-Stream Convolutional
Networks for Action Recognition in Videos. In NIPS, 2014.
1, 2, 4, 5, 6
[23] K. Simonyan and A. Zisserman. Very Deep Convolutional
Networks for Large-Scale Image Recognition. In ICLR,
2014. 3
[24] K. Soomro, A. R. Zamir, and M. Shah. UCF101: A Dataset
of 101 Human Action Classes From Videos in The Wild. In
CRCV-TR-12-01, 2012. 5
[25] C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed,
D. Anguelov, D. Erhan, V. Vanhoucke, and A. Rabinovich.
Going Deeper with Convolutions. In CVPR, 2015. 5
[26] D. Tran, L. Bourdev, R. Fergus, L. Torresani, and M. Paluri.
Learning Spatiotemporal Features with 3D Convolutional
Networks. In ICCV, 2015. 1, 2, 3, 5
[27] D. Tran, L. Bourdev, R. Fergus, L. Torresani, and M. Paluri.
Deep End2End Voxel2Voxel Prediction. In CVPRW DeepVision Workshop, 2016. 2, 3, 5
[28] C. Vondrick, H. Pirsiavash, and A. Torralba. Generating
videos with scene dynamics. In NIPS, 2016. 1, 2, 6
[29] J. Walker, C. Doersch, A. Gupta, and M. Hebert. An uncertain future: Forecasting from static images using variational
autoencoders. In ECCV, pages 835–851. Springer, 2016. 2,
3
[30] L. Wang, Y. Xiong, Z. Wang, Y. Qiao, D. Lin, X. Tang, and
L. Van Gool. Temporal segment networks: Towards good
practices for deep action recognition. In ECCV, 2016. 2
[31] X. Wang, A. Farhadi, and A. Gupta. Actions ˜ transformations. In CVPR, 2016. 2
[32] X. Wang and A. Gupta. Unsupervised learning of visual representations using videos. In ICCV, 2015. 2
[33] M. D. Zeiler and R. Fergus. Visualizing and Understanding
Convolutional Networks. In ECCV, 2014. 7

