Temporal Relational Reasoning in Videos

arXiv:1711.08496v1 [cs.CV] 22 Nov 2017

Bolei Zhou, Alex Andonian, Antonio Torralba
CSAIL, MIT

Abstract

?

Temporal relational reasoning, the ability to link meaningful transformations of objects or entities over time, is
a fundamental property of intelligent species. In this paper, we introduce an effective and interpretable network
module, the Temporal Relation Network (TRN), designed
to learn and reason about temporal dependencies between
video frames at multiple time scales. We evaluate TRNequipped networks on activity recognition tasks using three
recent video datasets - Something-Something, Jester, and
Charades - which fundamentally depend on temporal relational reasoning. Our results demonstrate that the proposed
TRN gives convolutional neural networks a remarkable capacity to discover temporal relations in videos. Through
only sparsely sampled video frames, TRN-equipped networks can accurately predict human-object interactions
in the Something-Something dataset and identify various
human gestures on the Jester dataset with very competitive performance. TRN-equipped networks also outperform
two-stream networks and 3D convolution networks in recognizing daily activities in the Charades dataset. Further
analyses show that the models learn intuitive and interpretable visual common sense knowledge in videos1 .

?
Figure 1. What takes place between two observations? (see answer below). Humans can easily infer the temporal relations and
transformations between these observations, but this task remains
difficult for neural networks.

timescales. For example, the activity of sprinting contains
the long-term temporal relations of crouching at the starting
blocks, running on track, and finishing at the end line, while
also includes the short-term temporal relations of periodic
hands and feet movement.
Activity recognition in videos has been one of the core
topics in computer vision. However, it remains difficult
due to the ambiguity of describing activities at appropriate timescales [21]. Many video datasets, such as UCF101
[24], Sport1M [11], and THUMOS [7], include many activities that can be recognized without reasoning about the
long-term temporal relations: still frames and optical flow
are sufficient to identify many of the labeled activities. Indeed, the classical two-stream Convolutional Neural Network [23] and the recent I3D Network [3], both based on
frames and optical flow, perform activity recognition very
well on these datasets.
However, convolutional neural networks still struggle in
situations where data and observations are limited, or where
the underlying structure is characterized by transformations
and temporal relations, rather than the appearance of certain entities [17, 14]. It remains remarkably challenging
for convolutional neural networks to reason about temporal relations and to anticipate what transformations are hap-

1. Introduction
The ability to reason about the relations between entities
over time is crucial for intelligent decision-making. Temporal relational reasoning allows intelligent species to analyze
the current situation relative to the past and formulate hypotheses on what may happen next. For example (Fig.1),
given two observations of an event, people can easily recognize the temporal relation between two states of the visual world and deduce what has happened between the two
frames of a video2 .
Temporal relational reasoning is critical for activity
recognition, forming the building blocks for describing the
steps of an event. A single activity can consist of several temporal relations at both short-term and long-term
1 Code

is available at http://relation.csail.mit.edu.
Poking a stack of cans so it collapses; Tidying up a closet

2 Answer:

1

pening to the observations. Fig.1 shows such examples.
The networks are required to discover visual common sense
knowledge over time beyond the appearance of objects in
the frames and the optical flow.
In this work, we propose a simple and interpretable network module called Temporal Relation Network (TRN) that
enables temporal relational reasoning in neural networks.
This module is inspired by the relational network proposed
in [17], but instead of modeling the spatial relations, TRN
aims to describe the temporal relations between observations in videos. Thus, TRN can learn and discover possible
temporal relations at multiple time scales. TRN is a general and extensible module that can be used in a plug-andplay fashion with any existing CNN architecture. We apply TRN-equipped networks on three recent video datasets
(Something-Something [8], Jester [1], and Charades [22]),
which are constructed for recognizing different types of activities such as human-object interactions and hand gestures, but all depend on temporal relational reasoning. The
TRN-equipped networks achieve very competitive results
even given only discrete RGB frames, bringing significant
improvements over baselines. Thus TRN provides a practical solution for standard neural networks to solve activity
recognition tasks using temporal relational reasoning.

1.1. Related Work
Convolutional Neural Networks for Activity Recognition. Activity recognition in videos is a core problem in computer vision. With the rise of deep convolutional neural networks (CNNs) which achieve state-of-theart performance on image recognition tasks [13, 29], many
works have looked into designing effective deep convolutional neural networks for activity recognition [11, 23, 4,
25, 28, 3]. For instance, various approaches of fusing RGB
frames over the temporal dimension are explored on the
Sport1M dataset [11]. Two stream CNNs with one stream
of static images and the other stream of optical flows are
proposed to fuse the information of object appearance and
short-term motions [23]. 3D convolutional networks [25]
use 3D convolution kernels to extract features from a sequence of dense RGB frames. Temporal Segment Networks sample frames and optical flow on different time segments to extract information for activity recognition [28].
A CNN+LSTM model, which uses a CNN to extract frame
features and an LSTM to integrate features over time, is also
used to recognize activities in videos [4]. Recently, I3D
networks [3] use two stream CNNs with inflated 3D convolutions on both dense RGB and optical flow sequences
to achieve state of the art performance on the Kinetics
dataset [12]. There are several important issues with existing CNNs for action recognition: 1) The dependency
on beforehand extraction of optical flow lowers the efficiency of the recognition system; 2) The 3D convolutions

on sequences of dense frames are computationally expensive, given the redundancy in consecutive frames; 3) Since
sequences of frames fed into the network are usually limited to 20 to 30 frames, it is difficult for the networks to
learn long-term temporal relations among frames. To address these issues, the proposed Temporal Relation Network
sparsely samples individual frames then learns their causal
relations, which is much more efficient than sampling dense
frames and convolving them. We show that TRN-equipped
networks can efficiently capture temporal relations at multiple time scales and outperform dense frame-based networks
using only sparsely sampled video frames.
Temporal Information in Activity Recognition. For
activity recognition on many existing video datasets such
as UCF101 [24], Sport1M [11], THUMOS [7], and Kinetics [12], the appearance of still frames and short-term motion such as optical flow are the most important information to identify the activities. Thus, activity recognition networks such as Two Stream network [23] and the I3D network [3] are tailored to capture these short-term dynamics
of dense frames. Therefore, existing networks don’t need to
build temporal relational reasoning abilities. On the other
hand, recently there are various video datasets collected via
crowd-sourcing, which focus on sequential activity recognition: Something-Something dataset [8] is collected for
generic human-object interaction. It has video classes such
as ‘Dropping something into something’, ‘Pushing something with something’, and even ‘Pretending to open something without actually opening it’. Jester dataset [1] is another recent video dataset for gesture recognition. Videos
are recorded by crowd-source workers performing 27 kinds
of gestures such as ‘Thumbing up’, ‘Swiping Left’, and
‘Turning hand counterclockwise’. Charades dataset is also
a high-level human activity dataset that collects videos by
asking crowd workers to perform a series of home activities and then record themselves [22]. For recognizing the
complex activities in these three datasets, it is crucial to integrate temporal relational reasoning into the networks. Besides, many previous works model the temporal structures
of videos for action recognition and detection using bag of
words, motion atoms, or action grammar [5, 16, 26, 6, 27].
Instead of designing temporal structures manually, we use
a more generic structure to learn the temporal relations in
end-to-end training. Thus it runs more efficiently to process
videos in testing.
Relational Reasoning and Intuitive Physics. Recently,
relational reasoning module has been proposed for visual
question answering with super-human performance [17].
Our work is inspired by that work, but we focus on modeling the multi-scale temporal relations in videos. In the domain of robot self-supervised learning, many models have
been proposed to learn the intuitive physics among frames.
Given an initial state and a goal state, the inverse dynam-

ics model with reinforcement learning is used to infer the
transformation between the object states [2]. Physical interaction and observations are also used to train deep neural
networks [15]. Time contrast networks are used for selfsupervised imitation learning of object manipulation from
third-person video observation [18]. Our work aims to learn
various temporal relations in videos in a supervised learning setting. The proposed TRN can be extended to selfsupervised learning for robot object manipulation.

Each relation term Td captures temporal relationships be(d)
tween d ordered frames. Each Td has its own separate hφ
(d)

and gθ . Notice that for any given sample of d frames for
each Td , all the temporal relation functions are end-to-end
differentiable, so they can all be trained together with the
base CNN used to extract features for each video frame.
The overall network framework is illustrated in Fig.2.

2.3. Efficient Training and Testing

2. Temporal Relation Networks
In this section, we introduce the framework of Temporal
Relation Networks. It is simple and can be easily plugged
into any existing convolutional neural network architecture
to enable temporal relational reasoning. In later experiments, we show that TRN-equipped networks discover interpretable visual common sense knowledge to recognize
activities in videos.

2.1. Defining Temporal Relations
Inspired by the relational reasoning module for visual
question answering [17], we define the pairwise temporal
relation as a composite function below:
T2 (V ) = hφ

X


gθ (fi , fj )

(1)

i<j

where the input is the video V with n selected ordered
frames as V = {f1 , f2 , ..., fn }, where fi is a representation of the ith frame of the video, e.g., the output activation
from some standard CNN. The functions hφ and gθ fuse
features of different ordered frames. Here we simply use
multilayer perceptrons (MLP) with parameters φ and θ respectively. For efficient computation, rather than adding all
the combination pairs, we uniformly sample frames i and j
and sort each pair.
We further extend the composite function of the 2-frame
temporal relations to higher frame relations such as the 3frame relation function below:
0

T3 (V ) = hφ

 X


0
gθ (fi , fj , fk )

(2)

i<j<k

where the sum is again over sets of frames i, j, k that have
been uniformly sampled and sorted.

2.2. Multi-Scale Temporal Relations
To capture temporal relations at multiple time scales, we
use the following composite function to accumulate frame
relations at different scales:
M TN (V ) = T2 (V ) + T3 (V )... + TN (V )

(3)

When training a multi-scale temporal network, we could
sample the sums by selecting different sets of d frames
for each Td term for a video. However, we use a sampling scheme that reduces computation significantly. First,
we uniformly sample a set of N frames from the video,
VN∗ ⊂ V , and we use VN∗ to calculate TN (V ). Then, for
each d < N , we choose k random subsamples of d frames
∗
⊂ VN∗ . These are used to compute the d-frame relations
Vkd
for each Td (V ). This allows O(kN 2 ) temporal relations to
be sampled while evaluating the CNN on only N frames.
At testing time, we can use the TRN-equipped network in a moving window to process long videos. A feature queue is used to cache the extracted CNN features of
equidistant frames sampled from the video, and those features are further combined into different relation tuples then
fed into TRN to predict the activity. The CNN features are
extracted from incoming frame only once then enqueued,
thus making TRN-equipped networks very efficient to run
in real-time on a desktop.

3. Experiments
We evaluate the TRN-equipped networks on a variety of
activity recognition tasks. For recognizing activities that depend on temporal relational reasoning, TRN-equipped networks outperform a baseline network without a TRN by a
large margin. We establish state-of-the-art results on the
Something-Something dataset for human-interaction recognition [8] and on the Jester dataset for hand gesture recognition [1]. The TRN-equipped networks also obtain competitive results on activity classification in the Charades
dataset [22], outperforming the Flow+RGB ensemble models [20, 22] using only sparsely sampled RGB frames.
The statistics of the three datasets Something-Something
dataset [8], Jester dataset [1], and Charades dataset [22] are
listed in Table 1. All three datasets are crowd-sourced, in
which the videos are collected by asking the crowd-source
workers to record themselves performing instructed activities. Unlike the Youtube-type videos in UCF101 and Kinetics, there is usually a clear start and end of each activity
in the crowd-sourced video, emphasizing the importance of
temporal relational reasoning.

1

2

Time
1

3

9

5

CNN

4

10

CNN

5

3

6

8

7

12

1

4

CNN

CNN

+

8

9

10

10

2

11

5

12

9

12

CNN

+

2-frame relation
+
3-frame relation
Pretending
to
put
something
next to something
4-frame relation
Figure 2. The illustration of Temporal Relation Networks. Representative frames of a video (shown above) are sampled and fed into
different frame relation modules. Only a subset of the 2-frame, 3-frame, and 4-frame relations are shown, as there are higher frame
relations included.
Table 1. Statistics of the datasets used in evaluating the TRNs.
Dataset
Something
Jester
Charades

Classes
174
27
157

Videos
108,499
148,092
9,848

Type
human-object interaction
human hand gesture
daily indoor activity

3.1. Network Architectures and Training
The networks used for extracting image features play
an important factor in visual recognition tasks [19]. Features from deeper networks such as ResNet [9] usually perform better. Our goal here is to evaluate the effectiveness
of the TRN module for temporal relational reasoning in
videos. Thus, we fix the base network architecture to be the
same throughout all the experiments and compare the performance of the CNN model with and without the proposed
TRN modules.
We adopt Inception with Batch Normalization (BNInception) pretrained on ImageNet used in [10] because of
its balance between accuracy and efficiency. We follow
the training strategies of partial BN (freezing all the batch
normalization layers except the first one) and dropout after
global pooling as used in [28]. We keep the network architecture of the MultiScale TRN module and the training
hyper-parameters the same for training models on all the
three datasets. We set k = 3 in the experiments as the number of accumulated relation triples in each relation module. gφ is simply a two-layer MLP with 256 units per layer,
while hφ is a one-layer MLP with the unit number matching
the class number. The CNN features for a given frame is the
activation from the BN-Inception’s global average pooling
layer (before the final classification layer). Given the BNInception as the base CNN, the training can be finished in

less than 24 hours for 100 training epochs on a single Nvidia
Titan Xp GPU. In the Multi-Scale TRN, we include all the
TRN modules from 2-frame TRN up to 8-frame TRN, as
including higher frame TRNs brings marginal improvement
and lowers the efficiency.

3.2. Results on Something-Something Dataset
Something-Something is a recent video dataset for
human-object interaction recognition. There are 174
classes, some of the ambiguous activity categories are challenging, such as ‘Tearing Something into two pieces’ versus
‘Tearing Something just a little bit’, ‘Turn something upside
down’ versus ‘Pretending to turn something upside down’.
We can see that the temporal relations and transformations
of objects rather than the appearance of the objects characterize the activities in the dataset.
The results on the validation set are listed in Table 2, in
which we compare the top1 and top5 accuracy for the base
network trained on single frames randomly selected from
each video, the base networks with various frame relation
modules, and the multi-scale TRN-equipped network. Networks with TRNs outperform the single frame baseline by
a large margin, while additional frames included in the relation bring further improvements. The Multi-Scale TRN
with ten crop data augmentation achieves the best performance.
We submit the prediction of the MultiScale TRN on the
test set to the official leaderboard3 . At the submission time,
our method tops the leaderboard using only discrete frames,
as shown in Table 3.
3 https://www.twentybn.com/datasets/something-something

Table 2. Results on the validation set of Something-Something
Dataset. All the models use the center cropping of the equidistant
frames in the video. Multi-scale TRN with 10-crop augmentation
achieves the best performance.

model
single frame
2-frame TRN
3-frame TRN
4-frame TRN
5-frame TRN
7-frame TRN
MultiScale TRN
MultiScale TRN (10-crop)

Top1 acc.(%)
11.41
22.23
26.22
29.83
30.39
31.01
33.01
34.44

Top5 acc.
33.39
48.80
54.15
58.21
58.29
59.24
61.27
63.20

Table 3. Results on the test set of the Something-Something
Dataset-V1. Comparison methods are from the official public
leaderboard.

model
Yana Hasson
Harrison.AI
I3D by [8]
Guillaume Berger
Besnet (Top1 on leaderboard)
MultiScale TRN

Top1 acc.(%)
25.55
26.38
27.23
30.48
31.66
33.60

Table 4. Results on the validation set of the Jester Dataset-V1.

model
single frame
2-frame TRN
MultiScale TRN
MultiScale TRN (10-crop)

Top1 acc.(%)
63.60
75.65
93.70
95.31

Top5 acc.
92.44
94.40
99.59
99.86

Table 5. Results on the test set of Jester dataset-V1.

model
20BN’s Jester System
VideoLSTM
Guillaume Berger
Ford’s Gesture Recognition System
Besnet (Top1 on leaderboard)
MultiScale TRN

Top1 acc.(%)
82.34
85.86
93.87
94.11
94.23
94.78

3.3. Results on Jester and Charades
We further evaluate the TRN-equipped networks on the
Jester dataset, which is a video dataset for hand gesture
recognition with 27 classes. The results on the validation
set of the Jester dataset are listed in Table 4. The result on
the test set and comparison with the top methods in the official leaderboard 4 are listed in Table 5. MultiScale TRN
again achieves the state-of-the-art performance with close
to 95% accuracy.
4 https://www.twentybn.com/datasets/jester

Table 6. Results on Charades Activity Classification.
Approach
Random
AlexNet
Two-Stream
MultiScale TRN

mAP
5.9
11.3
14.3
25.2

Approach
C3D
IDT
AsycTempField

mAP
10.9
17.2
22.4

We evaluate the MultiScale TRN on the recent Charades
dataset for daily activity recognition. The results are listed
in Table 6. Our method outperforms various methods such
as 2-stream networks and C3D [22], and the recent Asynchronous Temporal Field (AsycTempField) method [20].
The qualitative prediction results of the Multi-Scale TRN
on the three datasets are shown in Figure 3. The examples
in Figure 3 demonstrate that the TRN model is capable of
correctly identifying actions for which the overall temporal ordering of frames is essential for a successful prediction. For example, the turning hand counterclockwise category would assume a different class label when shown in
reverse. Moreover, the successful prediction of categories
in which an individual pretends to carry out an action (e.g.
‘pretending to put something into something’ as shown in
the second row) suggests that the network can capture temporal relations at multiple scales, where the ordering of several lower-level actions contained in short segments conveys
crucial semantic information about the overall activity class.
This outstanding performance shows the effectiveness of
the TRN for temporal relational reasoning and its strong
generalization ability across different datasets.

3.4. Interpreting Visual Common Sense Knowledge
inside the TRN
One of the distinct properties of the proposed TRNs
compared to previous video classification networks such as
C3D [25] and I3D [3] is that TRN has more interpretable
structure. In this section, we have a more in-depth analysis
to interpret the visual common sense knowledge learned by
the TRNs through solving these temporal reasoning tasks.
We explore the following four parts:
Representative frames of a video voted by the TRN
to recognize an activity. Intuitively, a human observer can
capture the essence of an action by selecting a small collection of representative frames. Does the same hold true
for models trained to recognize the activity? To obtain a
sequence of representative frames for each TRN, we first
compute the features of the equidistant frames from a video,
then randomly combine them to generate different frame relation tuples and pass them into the TRNs. Finally we rank
the relation tuples using the responses of different TRNs.
Figure 4 shows the top representative frames voted by different TRNs to recognize an activity in the same video. We
can see that the TRNs learn the temporal relations that characterize an activity. For comparatively simple actions, a
single frame is sufficient to establish some degree of con-

a Pouring something into something

Pushing something so that it almost falls oﬀ but doesn't

1: Pouring sth into sth (0.859). 2: Pouring sth into sth unAl it overﬂows (0.072)

1: Pushing sth so that it almost falls oﬀ but doesn't (0.280). 2: Pretending to take sth from sth (0.134)

Pretending to put something into something

Le9ng something roll along a ﬂat surface

1: Pretending to put sth into sth (0.710). 2: Failing to put sth into sth because sth does not ﬁt (0.265)

1: Rolling sth on a ﬂat surface (0.672). 2: LeHng sth roll along a ﬂat surface (0.192)

b Thumb Up

Turning Hand Counterclockwise

1: Thumb Up (0.999) 2: Thumb Down (0.001)

1: Turning Hand Counterclockwise (0.967) 2: Turning Hand Clockwise (0.033)

1: Zooming In With Two Fingers (0.993) 2: Zooming Out With Two Fingers (0.006)

1: Rolling Hand Forward (0.990) 2: Rolling Hand Backward (0.01)

Zooming In With Two Fingers

c

Holding a pillow (0.120)
Walking through a doorway (0.066)

Holding a pillow (0.188)
Holding a blanket (0.112)

Rolling Hand Backward

Holding a pillow (0.361)
Holding a blanket (0.083)

Taking a pillow from somewhere (0.207)
Holding a pillow (0.142)

Walking through a doorway (0.339), (0.245),
Holding a dish (0.109), Holding a book (0.131)

Holding a pillow (0.501), (0.401), (0.225)
Walking through a doorway (0.048), (0.100), (0.148)

Holding a pillow (0.118)
Taking a pillow from somewhere (0.079)

SiHng on the ﬂoor (0.267), (0.291), (0.229), (0.142)
Watching/Reading/Looking at a book (0.162), Holding a book (0.185), (0.163) (0.129)

Someone is sneezing (0.099)
SiHng in a chair (0.065)

Holding a book (0.163)
Closing a book (0.093)

Holding a pillow (0.151)
Lying on a bed (0.084)

Holding a dish (0.423)
Taking a dish(es) from
somewhere (0.154)

Lying on the ﬂoor (0.084), (0.124)
SiHng in a chair (0.071), ﬂoor (0.104)

Figure 3. Prediction examples on a) Something-Something, b) Jester, and c) Charades. For each example drawn from Something-Something
and Jester, the top two predictions and corresponding class probabilities accompany five uniformly-sampled frames with green text indicating a correct top1 prediction and red indicating an incorrect one. Due to the extended length of the videos in Charades, each plotted image
is the middle frame of a 10-frame sequence sampled evenly over a time interval, over which the top two class predictions are shown.

fidence in the correct action, but is vulnerable to mistakes
when a transformation is present. 2-frame TRN picks up
the two frames that best describe the transformation. Meanwhile, for more difficult activity categories such as ‘Pretending to poke something’, two frames are not sufficient information for even a human observer to differentiate. Similarly, the network needs additional frames in the TRNs to
correctly recognize the behavior.
Thus the progression of representative frames and their
corresponding class predictions inform us about how temporal relations may help the model reason about more complex behavior. One particular example is the last video in
Figure 4: The action’s context given by a single frame - a
hand close to a book - is enough to narrow down the top prediction to a qualitatively plausible action, unfolding something. A similar, two-frame relation marginally increases
the probability the initial prediction, although these two
frames would not be sufficient for even human observers to
make the correct prediction. Now, the three frame-relation
begins to highlight a pattern characteristic to Something-

Somethings set of pretending categories: the initial frames
closely resemble a certain action, but the later frames are
inconsistent with the completion of that action as if it never
happened. This relation helps the model to adjust its prediction to the correct class. Finally, the upward motion of the
individuals hand in the third frame of the 4-frame relation
further increases the discordance between the anticipated
and observed final state of the scene; a motion resembling
the action appeared to take place with no effect on the object, thus, solidifying confidence in the correct class prediction.
Importance of temporal order for activity recognition. To verify the importance of the temporal order of
frames for activity recognition, we conduct an experiment
to compare the scenario with input frames in temporal order and in shuffled order when training the TRNs, as shown
in Figure 5. For training the shuffled TRNs, we randomly
shuffle the frames in the relation modules. The significant difference on the Something-Something dataset shows
the importance of the temporal order in the activity recog-

2-frame TRN

Single-frame
5

0

3-frame TRN
5

1

4-frame TRN

3

5

0

1

4

5

2

4

7

1

4

7

Moving
sth down
Moving sth down (0.352) Moving sth down (0.998)

6

3

Moving sth down (0.999)

6

1

Moving sth down (0.999)

2

4

4

7

0

Covering
sth with sth
Uncovering sth (0.226)

6

Covering sth (0.997)

6

Covering sth (0.998)

7

0

Covering sth (0.999)

0

Throwing sth in the
air and catching it
Throwing sth in the air
and catching it (0.520)

Throwing sth in the air and catching it (0.986) Throwing sth in the air and catching it (0.999)

2

2

Unfolding sth (0.045)

Unfolding sth (0.164)

3

2

3

Throwing sth in the air and catching it (0.934)

6

2

3

4

5

Pretending to
open sth without
actually opening it
Pretending to open sth without actually opening it (0.828)

Pretending to open sth without actually opening it (0.870)

Figure 4. The top representative frames determined by single frame baseline network, the 2-frame TRN, 3-frame TRN, and 4-frame
TRN. TRNs learn to capture the essence of an activity only given a limited number of frames. Videos are from the validation set of the
Something-Something dataset
Something-Something

35
30
25
20
15
10

1

2

3

frame

4

UCF101

90

Ordered
Shuffled

Top1 Accuracy

Top1 Accuracy

nition. More interestingly, we repeat the same experiment on the UCF101 dataset [24] and observe no difference between the ordered frames and shuffled frames. That
shows activity recognition for the Youtube-type videos in
UCF101 don’t necessarily require the temporal reasoning
ability since there are not so many casual relations associated with an already on-going activity.
To further investigate how temporal ordering influences
activity recognition in TRN, we examine and plot the categories that show the largest differences in the class accuracy between ordered and shuffled inputs drawn from the
Something-Something dataset, in Figure 6. In general, actions with strong ‘directionality and large, one-way movements, such as ‘Moving something down’, appear to benefit the most from preserving the correct temporal ordering.
This observation aligns with the idea that the disruption of
continuous motion and a potential consequence of shuffling
video frames, would likely confuse a human observer, as it
would go against our intuitive notions of physics.
Interestingly, the penalty for shuffling frames of relatively static actions is less severe if penalizing at all in
some cases, with several categories marginally benefiting
from shuffled inputs, as observed with the category ‘putting
something that can’t roll onto a slanted surface so it stays
where it is’. Here, simply learning the coincidence of
frames rather than temporal transformations may be sufficient for the model to differentiate between similar activities and make the correct prediction.
Particularly in challenging ambiguous cases, for example ‘Pretending to throw something’ where the release point
is partially or completely obscured from view, disrupting
a strong ‘sense of motion’ may bias model predictions
away from the likely alternative, ‘throwing something’, frequently but incorrectly selected by the ordered model, thus

5

Ordered
Shuffled

85
80
75
70

1

2

3

4

5

frame

Figure 5. Accuracy obtained by different frame TRNs using ordered frames and shuffled frames, on Something-Something and
UCF101 dataset respectively. On Something-Something, the temporal order is critical for recognizing the activity. In contrast, recognizing activities in UCF101 does not necessarily require temporal relational reasoning.

Figure 6. The top 5 action categories that exhibited the largest
gain and the least gain (negative) respectively between ordered and
shuffled frames as inputs (difference value is labeled above). Actions with strong directional motion appear to suffer most from
shuffled inputs. In most cases, shuffling is least detrimental to actions with comparatively subtle movements.

giving rise to a curious difference in accuracy for that action.
t-SNE visualization of activity similarity. Figure 7
shows the t-SNE visualization for embedding the high-level

single frame

30

Uncovering sth

Tearing sth just a little bit

20

Tearing sth into two pieces

20

Moving sth closer to sth

0

0

Uncovering sth
Holding sth

-10

Covering sth with sth
Holding sth over sth

Plugging sth into sth
Showing sth to the camera

-20

Moving sth closer to sth

Tearing sth just a little bit
Tearing sth into two pieces

Pushing sth so that it almost falls off but doesn't
Covering sth with sth

Plugging sth into sth
Unfolding sth
Holding sth
Opening
Folding
sth sth
Holding sth over sth
Stuffing sth into sth
Sth falling like a feather or paper

20

Uncovering sth
Stuffing sth into sth
Holding sth
Holding sth over sth
Opening sth
Plugging sth into sth

Tearing sth into two pieces

-40

Folding sth
Sth falling like a feather or paper
Unfolding sth

0

-20

Sth falling like a feather or paper

-30

5-frame TRN
40

Showing sth to the camera

Pushing sth so that it almost falls off but doesn't
Stuffing sth into sth
Folding sth
Unfolding sth
Opening sth

10

-40

2-frame TRN

40

-20

Covering sth with sth

Pushing sth so that it almost falls
off but doesn't
Moving sth closer to sth

Tearing sth just a little bit

Showing sth to the camera

-40

-20

0

20

40

-60
-40

-40
-20

0

20

40

-20

0

20

40

Figure 7. t-SNE plot of the video samples of the 15 classes using the deep features from the single-frame baseline, 2-frame TRN, and
5-frame TRN. Higher frame TRN can better differentiate activities in Something-Something dataset (better view in color version with
zooming in).
First Frames

Forecasts

Ground Truth

1: Tearing sth just a liPle bit (0.998)
2: Tearing sth into two pieces (0.001)

Table 7. Anticipating activities using the MultiScale TRN on
Something-Something and Jester dataset. Only the first 25% and
50% of frames are given to the TRN to predict activities. Here the
baseline is the model trained on single frames.

3: Pretending to be tearing sth that is not tearable (0.001)
1: Li8ing a surface with sth on it but not enough for it to
slide down (0.490)

2: Li8ing sth with sth on it (0.423)
3: TilNng sth with sth on it slightly so it doesn't fall down (0.079)
1: Poking sth so lightly that it doesn't or almost doesn't move
(0.466)

Data
first 25%
first 50%
full

Something
baseline
TRN
9.08 11.14
10.10 19.10
11.41 33.01

Jester
baseline
TRN
27.25 34.23
41.43 78.42
63.60 93.70

2: Poking a stack of sth so the stack collapses (0.207)
3: Poking sth so it slightly moves (0.164)

1: Swiping Down (0.881)
2: Swiping Up (0.105)
3: Stop Sign (0.008)
1: Rolling Hand Forward (0.451)

2: Rolling Hand Backward (0.401)
3: No gesture (0.056)

Figure 8. Anticipating and forecasting activity when only given the
first 25% frames. The first 25% of each video, represented by the
first frame shown in the left column, is used to generate the top 3
anticipated forecasts and corresponding probabilities listed in the
middle column. The ground truth label is highlighted by a blue
arrow which points to the last frame of the video (not observed by
the network) on the right.

yet less explored problem in activity recognition. Here we
evaluate our TRN model on anticipating activity when given
only the first 25% and 50% of the frames in each validation
video. Results are shown in Table 7. For comparison, we
also include the single frame baseline. We see that TRN can
use the learned temporal relations to anticipate activity. The
performance increases as more ordered frames are received.
Figure 8 shows some examples of anticipating activities using only first 25% frames of a video. A qualitative review
of these examples reveals that model predictions on only
initial frames do serve as very reasonable forecasts despite
being given task with a high degree of uncertainty even for
human observers.

4. Conclusion
features from the single frame baseline, the 3-frame TRN,
and the 5-frame TRN, for the videos of the 15 most frequent activity classes in the validation set. We can see that
the features from 2-frame and 5-frame TRNs can better differentiate activity categories. We also observe the similarity among categories in the visualization map. For example,
‘Tearing something into two pieces’ is very similar to ‘Tearing something just a little bit’, and the categories ‘Folding
something’, ‘Unfolding something’, ‘Holding something’,
‘Holding something over something’ are clustered together.
Anticipating activities. Anticipating and forecasting activities before they happen or fully happen is a challenging

We proposed a simple and interpretable network module
called Temporal Relation Network (TRN) to enable temporal relational reasoning in neural networks for videos. We
evaluated the proposed TRN on several recent datasets and
established competitive results using only discrete frames.
Finally, we have shown that TRN modules discover visual
common sense knowledge in videos.
Acknowledgement: Project is supported by the Intelligence Advanced Research Projects Activity (IARPA) via
Department of Interior/ Interior Business Center (DOI/IBC)
contract number D17PC00344. The U.S. Government is
authorized to reproduce and distribute reprints for Govern-

mental purposes notwithstanding any copyright annotation
thereon. The first author thanks Jimmy Wu, Aude Oliva,
and Vadim Smolyakov for comments on the draft. BZ is
supported by Facebook Fellowship. Disclaimer: The views
and conclusions contained herein are those of the authors
and should not be interpreted as necessarily representing
the official policies or endorsements, either expressed or implied, of IARPA, DOI/IBC, or the U.S.

References
[1] Twentybn jester dataset:
a hand gesture dataset.
https://www.twentybn.com/datasets/jester, 2017.
[2] P. Agrawal, A. V. Nair, P. Abbeel, J. Malik, and S. Levine.
Learning to poke by poking: Experiential learning of intuitive physics. In Advances in Neural Information Processing
Systems, pages 5074–5082, 2016.
[3] J. Carreira and A. Zisserman. Quo vadis, action recognition? a new model and the kinetics dataset. arXiv preprint
arXiv:1705.07750, 2017.
[4] J. Donahue, L. Anne Hendricks, S. Guadarrama,
M. Rohrbach, S. Venugopalan, K. Saenko, and T. Darrell. Long-term recurrent convolutional networks for visual
recognition and description. In Proceedings of the IEEE
conference on computer vision and pattern recognition,
pages 2625–2634, 2015.
[5] A. Gaidon, Z. Harchaoui, and C. Schmid. Temporal localization of actions with actoms. IEEE transactions on pattern
analysis and machine intelligence, 35(11):2782–2795, 2013.
[6] A. Gaidon, Z. Harchaoui, and C. Schmid. Activity representation with motion hierarchies. International journal of
computer vision, 107(3):219–238, 2014.
[7] A. Gorban, H. Idrees, Y. Jiang, A. R. Zamir, I. Laptev,
M. Shah, and R. Sukthankar. Thumos challenge: Action
recognition with a large number of classes. In CVPR workshop, 2015.
[8] R. Goyal, S. Kahou, V. Michalski, J. Materzyńska, S. Westphal, H. Kim, V. Haenel, I. Fruend, P. Yianilos, M. MuellerFreitag, et al. The” something something” video database for
learning and evaluating visual common sense. Proc. ICCV,
2017.
[9] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages
770–778, 2016.
[10] S. Ioffe and C. Szegedy. Batch normalization: Accelerating
deep network training by reducing internal covariate shift. In
International Conference on Machine Learning, pages 448–
456, 2015.
[11] A. Karpathy, G. Toderici, S. Shetty, T. Leung, R. Sukthankar,
and L. Fei-Fei. Large-scale video classification with convolutional neural networks. In Proc. CVPR, 2014.
[12] W. Kay, J. Carreira, K. Simonyan, B. Zhang, C. Hillier, S. Vijayanarasimhan, F. Viola, T. Green, T. Back, P. Natsev, et al.
The kinetics human action video dataset. arXiv preprint
arXiv:1705.06950, 2017.

[13] A. Krizhevsky, I. Sutskever, and G. E. Hinton. Imagenet
classification with deep convolutional neural networks. In
Advances in neural information processing systems, pages
1097–1105, 2012.
[14] B. M. Lake, T. D. Ullman, J. B. Tenenbaum, and S. J. Gershman. Building machines that learn and think like people.
Behavioral and Brain Sciences, pages 1–101, 2016.
[15] L. Pinto, D. Gandhi, Y. Han, Y.-L. Park, and A. Gupta. The
curious robot: Learning visual representations via physical
interactions. In European Conference on Computer Vision,
pages 3–18. Springer, 2016.
[16] H. Pirsiavash and D. Ramanan. Parsing videos of actions
with segmental grammars. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages
612–619, 2014.
[17] A. Santoro, D. Raposo, D. G. Barrett, M. Malinowski,
R. Pascanu, P. Battaglia, and T. Lillicrap. A simple neural network module for relational reasoning. arXiv preprint
arXiv:1706.01427, 2017.
[18] P. Sermanet, C. Lynch, J. Hsu, and S. Levine. Timecontrastive networks: Self-supervised learning from multiview observation. arXiv preprint arXiv:1704.06888, 2017.
[19] A. Sharif Razavian, H. Azizpour, J. Sullivan, and S. Carlsson. Cnn features off-the-shelf: an astounding baseline for
recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition workshops, 2014.
[20] G. A. Sigurdsson, S. Divvala, A. Farhadi, and A. Gupta.
Asynchronous temporal fields for action recognition. 2017.
[21] G. A. Sigurdsson, O. Russakovsky, and A. Gupta. What actions are needed for understanding human actions in videos?
arXiv preprint arXiv:1708.02696, 2017.
[22] G. A. Sigurdsson, G. Varol, X. Wang, A. Farhadi, I. Laptev,
and A. Gupta. Hollywood in homes: Crowdsourcing data
collection for activity understanding. In European Conference on Computer Vision, pages 510–526. Springer, 2016.
[23] K. Simonyan and A. Zisserman. Two-stream convolutional
networks for action recognition in videos. In In Advances
in Neural Information Processing Systems, pages 568–576,
2014.
[24] K. Soomro, A. R. Zamir, and M. Shah. Ucf101: A dataset
of 101 human actions classes from videos in the wild. Proc.
CVPR, 2012.
[25] D. Tran, L. Bourdev, R. Fergus, L. Torresani, and M. Paluri.
Learning spatiotemporal features with 3d convolutional networks. In Proc. CVPR, 2015.
[26] H. Wang and C. Schmid. Action recognition with improved
trajectories. In Proc. ICCV, pages 3551–3558, 2013.
[27] L. Wang, Y. Qiao, and X. Tang. Mofap: A multi-level representation for action recognition. International Journal of
Computer Vision, 119(3):254–271, 2016.
[28] L. Wang, Y. Xiong, Z. Wang, Y. Qiao, D. Lin, X. Tang, and
L. Van Gool. Temporal segment networks: Towards good
practices for deep action recognition. In Proc. ECCV, pages
20–36. Springer, 2016.
[29] B. Zhou, A. Lapedriza, J. Xiao, A. Torralba, and A. Oliva.
Learning deep features for scene recognition using places
database. In Advances in neural information processing systems, pages 487–495, 2014.

