Learning to Generate Long-term Future via Hierarchical Prediction

Ruben Villegas 1 * Jimei Yang 2 Yuliang Zou 1 Sungryull Sohn 1 Xunyu Lin 3 Honglak Lee 1 4

arXiv:1704.05831v5 [] 8 Jan 2018

Abstract
We propose a hierarchical approach for making
long-term predictions of future frames. To avoid
inherent compounding errors in recursive pixellevel prediction, we propose to first estimate highlevel structure in the input frames, then predict
how that structure evolves in the future, and finally by observing a single frame from the past
and the predicted high-level structure, we construct the future frames without having to observe
any of the pixel-level predictions. Long-term
video prediction is difficult to perform by recurrently observing the predicted frames because the
small errors in pixel space exponentially amplify
as predictions are made deeper into the future.
Our approach prevents pixel-level error propagation from happening by removing the need to
observe the predicted frames. Our model is built
with a combination of LSTM and analogy-based
encoder-decoder convolutional neural networks,
which independently predict the video structure
and generate the future frames, respectively. In
experiments, our model is evaluated on the Human3.6M and Penn Action datasets on the task of
long-term pixel-level video prediction of humans
performing actions and demonstrate significantly
better results than the state-of-the-art.

1. Introduction
Learning to predict the future has emerged as an important research problem in machine learning and artificial
intelligence. Given the great progress in recognition (e.g.,
(Krizhevsky et al., 2012; Szegedy et al., 2015)), prediction becomes an essential module for intelligent agents to
plan actions or to make decisions in real-world application
scenarios (Jayaraman & Grauman, 2015; 2016; Finn et al.,
*

Work completed while at Google Brain. 1 Department of Electrical Engineering and Computer Science, University of Michigan,
Ann Arbor, MI, USA. 2 Adobe Research, San Jose, CA. 3 Beihang
University, Beijing, China. 4 Google Brain, Mountain View, CA.
Correspondence to: Ruben Villegas <rubville@umich.edu>.
Proceedings of the 34 th International Conference on Machine
Learning, Sydney, Australia, PMLR 70, 2017. Copyright 2017 by
the author(s).

2016). For example, robots can quickly learn manipulation
skills when predicting the consequences of physical interactions. Also, an autonomous car can brake or slow down
when predicting a person walking across the driving lane. In
this paper, we investigate long-term future frame prediction
that provides full descriptions of the visual world.
Recent recursive approaches to pixel-level video prediction
highly depend on observing the generated frames in the past
to make predictions further into the future (Oh et al., 2015;
Mathieu et al., 2016; Goroshin et al., 2015; Srivastava et al.,
2015; Ranzato et al., 2014; Finn et al., 2016; Villegas et al.,
2017; Lotter et al., 2017). In order to make reasonable longterm frame predictions in natural videos, these approaches
need to be highly robust to pixel-level noise. However, the
noise amplifies quickly through time until it overwhelms
the signal. It is common that the first few prediction steps
are of decent quality, but then the prediction degrades dramatically until all the video context is lost. Other existing
works focus on predicting high-level semantics, such as trajectories or action labels (Walker et al., 2014; Chao et al.,
2017; Yuen & Torralba, 2010; Lee, 2015), driven by immediate applications (e.g., video surveillance). We note
that such high-level representations are the major factors for
explaining the pixel variations into the future. In this work,
we assume that the high-dimensional video data is generated from low-dimensional high-level structures, which we
hypothesize will be critical for making long-term visual predictions. Our main contribution is the hierarchical approach
for video prediction that involves generative modeling of
video using high-level structures. Concretely, our algorithm
first estimates high-level structures of observed frames, and
then predicts their future states, and finally generates future
frames conditioned on predicted high-level structures.
The prediction of future structure is performed by an LSTM
that observes a sequence of structures estimated by a CNN,
encodes the observed dynamics, and predicts the future sequence of such structures. We note that Fragkiadaki et al.
(2015) developed an LSTM architecture that can straightforwardly be adapted to our method. However, our main
contribution is the hierarchical approach for video prediction, so we choose a simpler LSTM architecture to convey our idea. Our approach then observes a single frame
from the past and predicts the entire future described by
the predicted structure sequence using an analogy-making

Learning to Generate Long-term Future via Hierarchical Prediction

network (Reed et al., 2015). In particular, we propose an
image generator that learns a shared embedding between
image and high-level structure information which allows us
convert an input image into a future image guided by the
structure difference between the input image and the future
image. We evaluate the proposed model on challenging
real-world human action video datasets. We use 2D human
poses as our high-level structures similar to Reed et al.
(2016a). Thus, our LSTM network models the dynamics of
human poses while our analogy-based image generator network learns a joint image-pose embedding that allows the
pose difference between an observed frame and a predicted
frame to be transferred to image domain for future frame
generation. As a result, this pose-conditioned generation
strategy prevents our network from propagating prediction
errors through time, which in turn leads to very high quality
future frame generation for long periods of time. Overall,
the promising results of our approach suggest that it can be
greatly beneficial to incorporate proper high-level structures
into the generative process.
The rest of the paper is organized as follows: A review of the
related work is presented in Section 2. The overview of the
proposed algorithm is presented in Section 3. The network
configurations and their training algorithms are described
in Section 4 and Section 5, respectively. We present the
experimental details and results in Section 6, and conclude
the paper with discussions of future work in Section 7.

2. Related Work
Early work on future frame prediction focused on small
patches containing simple predictable motions (Sutskever
et al., 2009; Michalski et al., 2014; Mittelman et al., 2014)
and motions in real videos (Ranzato et al., 2014; Srivastava et al., 2015). High resolution videos contain far more
complicated motion which cannot be modeled in a patchwise manner due to the well known aperture problem. The
aperture problem causes blockiness in predictions as we
move forward in time. Ranzato et al. (2014) tried to solve
blockiness by averaging over spatial displacements after
predicting patches; however, this approach does not work
for long-term predictions.
Recent approaches in video prediction have moved from predicting patches to full frame prediction. Oh et al. (2015) proposed a network architecture for action conditioned video
prediction in Atari games. Mathieu et al. (2016) proposed
an adversarial loss for video prediction and a multi-scale
network architecture that results in high quality prediction
for a few timesteps in natural video; however, the frame prediction quality degrades quickly. Finn et al. (2016) proposed
a network architecture to directly transform pixels from a
current frame into the next frame by predicting a distribution
over pixel motion from previous frames. Xue et al. (2016)
proposed a probabilistic model for predicting possible mo-

tions of a single input frame by training a motion encoder in
a variational autoencoder approach. Vondrick et al. (2016)
built a model that generates realistic looking video by separating background and foreground motion. Villegas et al.
(2017) improved the convolutional encoder/decoder architecture by separating motion and content features. Lotter
et al. (2017) built an architecture inspired by the predictive coding concept in neuroscience literature that predicts
realistic looking frames.
All the previously mentioned approaches attempt to perform
video generation in a pixel-to-pixel process. We aim to
perform the prediction of future frames in video by taking
a hierarchical approach of first predicting the high-level
structure and then using the predicted structure to predict
the future in the video from a single frame input.
To the best of our knowledge, this is the first hierarchical
approach to pixel-level video prediction. Our hierarchical
architecture makes it possible to generate good quality longterm predictions that outperform current approaches. The
main success from our algorithm comes from the novel idea
of first making high-level structure predictions which allows
us to observe a single image and generate the future video
by visual-structure analogy. Our image generator learns
a shared embedding between image and structure inputs
that allows us to transform high-level image features into a
future image driven by the predicted structure sequence.

3. Overview
This paper tackles the task of long-term video prediction in
a hierarchical perspective. Given the input high-level structure sequence p1:t and frame xt , our algorithm is asked to
predict the future structure sequence pt+1:t+T and subsequently generate frames xt+1:t+T . The problem with video
frame prediction originates from modeling pixels directly
in a sequence-to-sequence manner and attempting to generate frames in a recurrent fashion. Current state-of-the-art
approaches recurrently observe the predicted frames, which
causes rapidly increasing error accumulation through time.
Our objective is to avoid having to observe generated future
frames at all during the full video prediction procedure.
Figure 1 illustrates our hierarchical approach. Our full
pipeline consists of 1) performing high-level structure estimation from the input sequence, 2) predicting a sequence of
future high-level structures, and 3) generating future images
from the predicted structures by visual-structure analogymaking given an observed image and the predicted structures. We explore our idea by performing pixel-level video
prediction of human actions while treating human pose as
the high-level structure. Hourglass network (Newell et al.,
2016) is used for pose estimation on input images. Subsequently, a sequence-to-sequence LSTM-recurrent network
is trained to read the outputs of hourglass network and to

Learning to Generate Long-term Future via Hierarchical Prediction
Pose
Estimation

Pose
Predic
tion

Image
Generation

t=1
t=2
t=3

t=5

t=5

t=6

t=6

t=7

t=7

t=8
t=9
t=10

t=8
t=9
t=10

t=4

Figure 1. Overall hierarchical approach to pixel-level video prediction. Our algorithm first observes frames from the past and estimate the
high-level structure, in this case human pose xy-coordinates, in each frame. The estimated structure is then used to predict the future
structures in a sequence to sequence manner. Finally, our algorithm takes the last observed frame, its estimated structure, and the predicted
structure sequence, in this case represented as heatmaps, and generates the future frames. Green denotes input to our network and red
denotes output from our network.

predict the future pose sequence. Finally, we generate the
future frames by analogy making using the pose relationship
in feature space to transform the last observed frame.
The proposed algorithm makes it possible to decompose
the task of video frame prediction to sub-tasks of future
high-level structure prediction and structure-conditioned
frame generation. Therefore, we remove the recursive dependency of generated frames that causes the compound
errors of pixel-level prediction in previous methods, and so
our method performs very long-term video prediction.

4. Architecture
This section describes the architecture of the proposed algorithm using human pose as a high-level structure. Our
full network is composed of two modules: an encoderdecoder LSTM that observes and outputs xy-coordinates,
and an image generator that performs visual analogy based
on high-level structure heatmaps constructed from the xycoordinates output from LSTM.
4.1. Future Prediction of High-Level Structures
Figure 2 illustrates our pose predictor. Our network first
encodes the observed structure dynamics by
[ht , ct ] = LSTM (pt , ht−1 , ct−1 ) ,

(1)

where ht ∈ RH represents the observed dynamics up to
time t, ct ∈ RH is the memory cell that retains information
from the history of pose inputs, pt ∈ R2L is the pose at
time t (i.e., 2D coordinate positions of L joints). In order
to make a reasonable prediction of the future pose, LSTM
has to first observe a few pose inputs to identify the type of
motion occurring in the pose sequence and how it is changing over time. LSTM also has to be able to remove noise
present in the input pose, which can come from annotation
error if using the dataset-provided pose annotation or pose
estimation error if using a pose estimation algorithm. After
a few pose inputs have been observed, LSTM generates the
future pose by

p̂t = f w> ht ,
(2)

LSTM

LSTM

LSTM

LSTM

LSTM

LSTM

Figure 2. Illustration of our pose predictor. LSTM observes k
consecutive human pose inputs and predicts the pose for the next
T timesteps. Note that the human heatmaps are used for illustration
purposes, but our network observes and outputs xy-coordinates.

where w is a projection matrix, f is a function on the projection (i.e. tanh or identity), and p̂t ∈ R2L is the predicted
pose. In the subsequent predictions, our LSTM does not
observe the previously generated pose. Not observing generated pose in LSTM prevents errors in the pose prediction
from being propagated into the future, and it also encourages the LSTM internal representation to contain robust
high-level features that allow it to generate the future sequence from only the original observation. As a result, the
representation obtained in the pose input encoding phase
must obtain all the necessary information for generating the
correct action sequence in the decoding phase. After we
have set the human pose sequence for the future frames, we
proceed to generate the pixel-level visual future.
4.2. Image Generation by Visual-Structure Analogy
To synthesize a future frame given its pose structure, we
make a visual-structure analogy inspired by Reed et al.
(2015) following pt : pt+n :: xt : xt+n , read as "pt is to
pt+n as xt is to xt+n " as illustrated in Figure 3. Intuitively,
the future frame xt+n can be generated by transferring the
structure transformation from pt to pt+n to the observed
frame xt . Our image generator instantiates this idea using
a pose encoder fpose , an image encoder fimg and an image
decoder fdec . Specifically, fpose is a convolutional encoder
that specializes on identifying key pose features from the

Learning to Generate Long-term Future via Hierarchical Prediction
t=5

t=40

:

t=5

t=40

:

::

?

Figure 3. Generating image frames by making analogies between
high-level structures and image pixels.

ƒpose
pt+n

_

ƒdec

ƒpose

Algorithm 1 Video Prediction Procedure
input: x1:k
output: x̂k+1:k+T
for t=1 to k do
pt ← Hourglass(xt )
[ht , ct ] ← LSTM(pt , ht−1 , ct−1 )
end for
for t=k + 1 to k + T do
[ht , ct ] ← LSTM(h
 t−1 , ct−1 )
p̂t ← f w> ht
x̂t ← fdec (fpose (g (p̂t )) − fpose (g (pk )) + fimg (xk ))
end for

+

pt

ƒimg

xt+n

xt
Figure 4. Illustration of our image generator. Our image generator
observes an input image, its corresponding human pose, and the
human pose of the future image. Through analogy making, our
network generates the next frame.

pose input that reflects high-level human structure.1 fimg is
also a convolutional encoder that acts on an image input by
mapping the observed appearance into a feature space where
the pose feature transformations can be easily imposed to
synthesize the future frame using the convolutional decoder
fdec . The visual-structure analogy is then performed by
x̂t+n = fdec (fpose (g (p̂t+n )) − fpose (g (pt )) + fimg (xt )) ,
(3)
where x̂t+n and p̂t+n are the generated image and corresponding predicted pose at time t + n, xt and pt are the
input image and corresponding estimated pose at time t, and
g (.) is a function that maps the output xy-coordinates from
LSTM into L depth-concatenated heatmaps.2 Intuitively,
fpose infers features whose “substractive" relationship is
the same subtractive relationship between xt+n and xt in
the feature space computed by fimg , i.e., fpose (g(p̂t+n )) −
fpose (g(p̂t )) ≈ fimg (xt+n ) − fimg (xt ). The network diagram is illustrated in in Figure 4. The relationship discovered by our network allows for highly non-linear transformations between images to be inferred by a simple addition/subtraction in feature space.

5. Training
In this section, we first summarize the multi-step video
prediction algorithm using our networks and then describe
1
Each input pose to our image generator is converted to concatenated heatmaps of each landmark before computing features.
2
We independently construct the heatmap with a Gaussian
function around the xy-coordinates of each landmark.

the training strategies of the high-level structure LSTM and
of the visual-structure analogy network. We train our highlevel structure LSTM independent from the visual-structure
analogy network, but both are combined during test time to
perform video prediction.
5.1. Multi-Step Prediction
Our algorithm multi-step video prediction procedure is described in Algorithm 1. Given input video frames, we use
the Hourglass network (Newell et al., 2016) to estimate
the human poses p1:k . High-level structure LSTM then
observes p1:k , and proceeds to generate a pose sequence
p̂k+1:k+T where T is the desired number of time steps to
predict. Next, our visual-structure analogy network takes xk ,
pk , and p̂k+1:k+T and proceeds to generate future frames
x̂k+1:k+T one by one. Note that the future frame prediction
is performed by observing pixel information from only xk ,
that is, we never observe any of the predicted frames.
5.2. High-Level Structure LSTM Training
We employ a sequence-to-sequence approach to predict
the future structures (i.e. future human pose). Our LSTM
is unrolled for k timesteps to allow it to observe k pose
inputs before making any prediction. Then we minimize the
prediction loss defined by
T

Lpose =

L

1 XX
1{mlk+t =1} kp̂lk+t − plk+t k22 ,
T L t=1

(4)

l=1

where p̂lk+t and plk+t are the predicted and ground-truth
pose l-th landmark, respectively, 1{.} is the indicator function, and mlk+t tells us whether a landmark is visible or
not (i.e. not present in the ground-truth). Intuitively, the
indicator function allows our LSTM to make a guess of the
non-visible landmarks even when not present at training.
Even in the absence of a few landmarks during training,
LSTM is able to internally understand the human structure
and observed motion. Our training strategy allows LSTM to
make a reasonable guess of the landmarks not present in the
training data by using the landmarks available as context.

Learning to Generate Long-term Future via Hierarchical Prediction

5.3. Visual-Structure Analogy Training
Training our network to transform an input image into a
target image that is too close in image space can lead to suboptimal parameters being learned due to the simplicity of
such task that requires only changing a few pixels. Because
of this, we train our network to perform random jumps in
time within a video clip. Specifically, we let our network
observe a frame xt and its corresponding human pose pt ,
and force it to generate frame xt+n given pose pt+n , where
n is defined randomly for every iteration at training time.
Training to jump to random frames in time gives our network a clear signal the task at hand due to the large pixel
difference between frames far apart in time.
To train our network, we use the compound loss from Dosovitskiy & Brox (2016). Our network is optimized to minimize the objective given by
L = Limg + Lfeat + LGen ,

(5)

where Limg is the loss in image space defined by
Limg = kxt+n − x̂t+n k22 ,

(6)

where xt+n and x̂t+n are the target and predicted frames,
respectively. The image loss intuitively guides our network
towards a rough blurry pixel-leven frame prediction that
reflects most details of the target image. Lfeat is the loss in
feature space define by
Lfeat = kC1 (xt+n ) − C1 (x̂t+n ) k22
+ kC2 (xt+n ) − C2 (x̂t+n ) k22 ,

(7)

where C1 (.) extracts features representing mostly image
appearance, and C2 (.) extracts features representing mostly
image structure. Combining appearance sensitive features
with structure sensitive features gives our network a learning
signal that allows it to make frame predictions with accurate
appearance while also enforcing correct structure. LGen is
the term in adversarial loss that allows our model to generate
realistic looking images and is defined by
LGen = − log D ([pt+n , x̂t+n ]) ,

(8)

where x̂t+n is the predicted image, pt+n is the human pose
corresponding to the target image, and D (.) is the discriminator network in adversarial loss. This sub-loss allows our
network to generate images that reflect a similar level of
detail as the images observed in the training data.
During the optimization of D, we use the mismatch term
proposed by Reed et al. (2016b), which allows the discriminator D to become sensitive to mismatch between the generation and the condition. The discriminator loss is defined
by
LDisc = − log D ([pt+n , xt+n ])
− 0.5 log (1 − D ([pt+n , x̂t+n ]))
− 0.5 log (1 − D ([pt+n , xt ])) ,

(9)

while optimizing our generator with respect to the adversarial loss, the mismatch-aware term sends a stronger signal to
our generator resulting in higher quality image generation,
and network optimization. Essentially, having a discriminator that knows the correct structure-image relationship,
reduces the parameter search space of our generator while
optimizing to fool the discriminator into believing the generated image is real. The latter in combination with the other
loss terms allows our network to produce high quality image
generation given the structure condition.

6. Experiments
In this section, we present experiments on pixel-level
video prediction of human actions on the Penn Action
(Weiyu Zhang & Derpanis, 2013) and Human 3.6M datasets
(Ionescu et al., 2014). Pose landmarks and video frames
are normalized to be between -1 and 1, and frames are
cropped based on temporal tubes to remove as much background as possible while making sure the human of interest is in all frames. For the feature similarity loss term
(Equation 7), we use we use the last convolutional layer
in AlexNet (Krizhevsky et al., 2012) as C1 , and the last
layer of the Hourglass Network in Newell et al. (2016) as
C2 . We augmented the available video data by performing horizontal flips randomly at training time for Penn
Action. Motion-based pixel-level quantitative evaluation
using Peak Signal-to-Noise Ratio (PSNR), analysis, and
control experiments can be found in the supplementary material. For video illustration of our method, please refer to
the project website: https://sites.google.com/
a/umich.edu/rubenevillegas/hierch_vid.
We compare our method against two baselines based on
convolutional LSTM and optical flow. The convolutional
LSTM baseline (Shi et al., 2015) was trained with adversarial loss (Mathieu et al., 2016) and the feature similarity loss
(Equation 7). An optical flow based baseline used the last
observed optical flow (Farneback, 2003) to move the pixels
of the last observed frame into the future.
We follow a human psycho-physical quantitative evaluation
metric similar to Vondrick et al. (2016). Amazon Mechanical Turk (AMT) workers are given a two-alternative choice
to indicate which of two videos looks more realistic. Specifically, the workers are shown a pair of videos (generated by
two different methods) consisting of the same input frames
indicated by a green box and predicted frames indicated
by a red box, in addition to the action label of the video.
The workers are instructed to make their decision based on
the frames in the red box. Additionally, we train a Twostream action recognition network (Simonyan & Zisserman,
2014) on the Penn Action dataset and test on the generated
videos to evaluate if our network is able to generate videos
predicting the activities observed in the original dataset.
We do not perform action classification experiments on the

Learning to Generate Long-term Future via Hierarchical Prediction
Method
Temporal Stream Spatial Stream Combined
Real Test Data *
66.6%
63.3%
72.1%
Ours
35.7%
52.7%
59.0%
Convolutional LSTM
13.9%
45.1%
46.4%
Optical Flow
13.9%
39.2%
34.9%
Table 1. Activity recognition evaluation.
"Which video is more realistic?"
Baseball Clean & jerk Golf Jumping jacks Jump rope Tennis Mean
Prefers ours over Convolutional LSTM 89.5%
87.2%
84.7%
83.0%
66.7%
88.2% 82.4%
Prefers ours over Optical Flow
87.8%
86.5%
80.3%
88.9%
86.2%
85.6% 86.1%
Table 2. Penn Action Video Generation Preference: We show videos from two methods to Amazon Mechanical Turk workers and ask
them to indicate which is more realistic. The table shows the percentage of times workers preferred our model against baselines. A
majority of the time workers prefer predictions from our model. We merged baseball pitch and baseball swing into baseball, and tennis
forehand and tennis serve into tennis.

Human3.6M dataset due to high uncertainty in the human
movements and high motion similarity amongst actions.
Architectures. The sequence prediction LSTM is made of
a single layer encoder-decoder LSTM with tied parameters,
1024 hidden units, and tanh output activation. Note that
the decoder LSTM does not observe any inputs other than
the hidden units from the encoder LSTM as initial hidden
units. The image and pose encoders are built with the same
architecture as VGG16 (Simonyan & Zisserman, 2015) up to
the third pooling layer, except that the pose encoder takes in
the pose heat-maps as an image made of L channels, and the
image encoder takes a regular 3-channel image. The decoder
is the mirrored architecture of the image encoder where
we perform unpooling followed by deconvolution, and a
final tanh activation. The convolutional LSTM baseline is
built with the same architecture as the image encoder and
decoder, but there is a convolutional LSTM layer with the
same kernel size and number of channels as the last layer in
the image encoder connecting them.
6.1. Penn Action Dataset
Experimental setting. The Penn Action dataset is composed of 2326 video sequences of 15 different actions and
13 human joint annotations for each sequence. To train our
image generator, we use the standard train split provided in
the dataset. To train our pose predictor, we sub-sample the
actions in the standard train-test split due to very noisy joint
ground-truth. We used videos from the actions of baseball
pitch, baseball swing, clean and jerk, golf swing, jumping
jacks, jump rope, tennis forehand, and tennis serve. Our
pose predictor is trained to observe 10 inputs and predict 32
steps, and tested on predicting up to 64 steps (some videos’
groundtruth end before 64 steps). Our image generator is
trained to make single random jumps within 30 steps into
the future. Our evaluations are performed on a single clip
that starts at the first frame of each video.
AMT results. These experiments were performed by 66
unique workers, where a total of 1848 comparisons were
made (934 against convolutional LSTM and 914 against op-

tical flow baseline). As shown in Table 2 and Figure 5, our
method is capable of generating more realistic sequences
compared to the baselines. Quantitatively, the action sequences generated by our network are perceptually higher
quality than the baselines and also predict the correct action
sequence. A relatively small (although still substantial) margin is observed when comparing to convolutional LSTM
for the jump rope action (i.e., 66.7% for ours vs 33.3% for
Convolutional LSTM). We hypothesize that convolutional
LSTM is able to do a reasonable job for this action class due
the highly cyclic motion nature of jumping up and down in
place. The remainder of the human actions contain more
complicated non-linear motion, which is much more complicated to predict. Overall, our method outperforms the
baselines by a large margin (i.e. 82.4% for ours vs 17.6%
for Convolutional LSTM, and 86.1% for ours vs 13.9% for
Optical Flow). Side by side video comparison for all actions
can be found in our project website.
Action recognition results. To see whether the generated
videos contain actions that can fool a CNN trained for action
recognition, we train a Two-Stream CNN on the PennAction dataset. In Table 1, “Temporal Stream” denotes the
network that observes motion as concatenated optical flow
(Farneback’s optical flow) images as input, and “Spatial
Stream” denotes the network that observes single image as
input. “Combined” denotes the averaging of the output probability vectors from the Temporal and Spatial stream. “Real
test data” denotes evaluation on the ground-truth videos (i.e.
perfect prediction).
From Table 1, it is shown that our network is able to generate
videos that are far more representative of the correct action
compared to all baselines, in both Temporal and Spatial
stream, regardless of using a neural network as the judge.
When combining both Temporal and Spatial streams, our
network achieves the best quality videos in terms of making
a pixel-level prediction of the correct action.
Pixel-level evaluation and control experiments. We
evaluate the frames generated by our method using PSNR

Learning to Generate Long-term Future via Hierarchical Prediction
"Which video is more realistic?"
Directions Discussion Eating Greeting Phoning
Prefers ours over Convolutional LSTM
67.6%
75.9%
74.7% 79.5% 69.7%
Prefers ours over Optical Flow
61.4%
89.3%
43.8% 80.3% 84.5%
"Which video is more realistic?"
Purchases Sitting Sittingdown Smoking Waiting
Prefers ours over Convolutional LSTM 79.0% 38.0%
54.7%
70.4% 50.0%
Prefers ours over Optical Flow
85.7% 35.1%
46.7%
73.3% 84.3%

Photo Posing
66.2% 69.7%
52.0% 75.3%
Walking Mean
86.0% 70.3%
90.8% 72.3%

Table 3. Human3.6M Video Generation Preference: We show videos from two methods to Amazon Mechanical Turk workers and ask
them to indicate which of the the two looks more realistic. The table shows the percentage of times workers preferred our model against
baselines. Most of the time workers prefer predictions from our model. We merge the activity categories of walking, walking dog, and
walking together into walking.

as measure, and separated the test data based on amount
of motion, as suggested by Villegas et al. (2017). From
these experiments, we conclude that pixel-level evaluation
highly depends on predicting the exact future observed in
the ground-truth. Highest PSNR scores are achieved when
trajectories of the exact future is used to generate the future frames. Due to space constraints, we ask the reader to
please refer to the supplementary material for more detailed
quantitative and qualitative analysis.
6.2. Human3.6M Dataset
Experimental settings. The Human3.6M dataset
(Ionescu et al., 2014) is composed of 3.6 million 3D human
poses (we use the provided 2D pose projections) composed
of 32 joints and corresponding images taken from 11
professional actors in 17 scenarios. For training, we use
subjects number 1, 5, 6, 7, and 8, and test on subjects
number 9 and 11. Our pose predictor is trained to observe
10 inputs and predict 64 steps, and tested on predicting
128 steps. Our image generator is trained to make single
random jumps anywhere in the training videos. We evaluate
on a single clip from each test video that starts at the exact
middle of the video to make sure there is motion occurring.
AMT results. We collected a total of 2203 comparisons
(1086 against convolutional LSTM and 1117 against optical
flow baseline) from 71 unique workers. As shown in Table 3, the videos generated by our network are perceptually
higher quality and reflect a reasonable future compared to
the baselines on average. Unexpectedly, our network does
not perform well on videos where the action involves minimal motion, such as sitting, sitting down, eating, taking a
photo, and waiting. These actions usually involve the person
staying still or making very unnoticeable motion which can
result in a static prediction (by convolutional LSTM and/or
optical flow) making frames look far more realistic than the
prediction from our network. Overall, our method outperforms the baselines by a large margin (i.e. 70.3% for ours
vs 29.7% for Convolutional LSTM, and 72.3% for ours vs
27.7% for Optical Flow). Figure 5 shows that our network
generates far higher quality future frames compared to the
convolutional LSTM baseline. Side by side video comparison for all actions can be found in our project website.

Pixel-level evaluation and control experiments. Following the same procedure as Section 6.1, we evaluated
the predicted videos using PSNR and separated the test data
by motion. Due to the high uncertainty and number of prediction steps in these videos, the predicted future can largely
deviate from the exact future observed in the ground-truth.
The highest PSNR scores are again achieved when the exact
future pose is used to generate the video frames; however,
there is an even larger gap compared to the results in Section 6.1. Due to space constraints, we ask the reader to
please refer to the supplementary material for more detailed
quantitative and qualitative analysis.

7. Conclusion and Future Work
We propose a hierarchical approach of pixel-level video
prediction. Using human action videos as benchmark, we
have demonstrated that our hierarchical prediction approach
is able to predict up to 128 future frames, which is an order
of magnitude improvement in terms of effective temporal
scale of the prediction.
The success of our approach demonstrates that it can be
greatly beneficial to incorporate the proper high-level structure into the generative process. At the same time, an important open research question would be how to automatically
learn such structures without domain knowledge. We leave
this as future work.
Another limitation of this work is that it generates a single
future trajectory. For an agent to make a better estimation
of what the future looks like, we would need more than one
generated future. Future work will involve the generation of
many futures given using a probabilistic sequence model.
Finally, our model does not handle background motion. This
is a highly challenging task since background comes in and
out of sight. Predicting background motion will require a
generative model that hallucinates the unseen background.
We also leave this as future work.

Acknowledgments
This work was supported in part by ONR N00014-13-10762, NSF CAREER IIS-1453651, Gift from Bosch Research, and Sloan Research Fellowship. We thank NVIDIA
for donating K40c and TITAN X GPUs.

Learning to Generate Long-term Future via Hierarchical Prediction
t=20

t=29

t=38

t=47

t=56

t=65

t=11

t=29

t=47

t=65

t=83

t=101

t=119

Input frames Groundtruth Conv LSTM

Predicted
frames (ours)

Predicted
pose (ours)

Input frames Groundtruth Conv LSTM

Predicted
frames (ours)

Predicted
pose (ours)

t=11

Figure 5. Qualitative evaluation of our network for 55 step prediction on Penn Action (top rows), and 109 step prediction on Human3.6M
(bottom rows). Our algorithm observes 10 previous input frames, estimates the human pose, predicts the pose sequence of the future, and
it finally generates the future frames. Green box denotes input and red box denotes prediction. We show the last 7 input frames. Side by
side video comparisons can be found in our project website.

Learning to Generate Long-term Future via Hierarchical Prediction

References
Chao, Y.-W., Yang, J., Price, B., Cohen, S., and Deng,
J. Forecasting human dynamics from static images. In
CVPR, 2017. 1
Dosovitskiy, A. and Brox, T. Generating images with perceptual similarity metrics based on deep networks. In
NIPS, 2016. 5
Farneback, G. Two-frame motion estimation based on polynomial expansion. In SCIA, 2003. 5

Newell, A., Yang, K., and Deng, J. Stacked hourglass
networks for human pose estimation. In ECCV. 2016. 2,
4, 5
Oh, J., Guo, X., Lee, H., Lewis, R. L., and Singh, S. Actionconditional video prediction using deep networks in atari
games. In NIPS. 2015. 1, 2
Ranzato, M., Szlam, A., Bruna, J., Mathieu, M., Collobert, R., and Chopra, S. Video (language) modeling: a
baseline for generative models of natural videos. arXiv
preprint:1412.6604, 2014. 1, 2

Finn, C., Goodfellow, I. J., and Levine, S. Unsupervised
learning for physical interaction through video prediction.
In NIPS. 2016. 1, 2

Reed, S., Zhang, Y., Zhang, Y., and Lee, H. Deep visual
analogy-making. In NIPS. 2015. 2, 3

Fragkiadaki, K., Levine, S., Felsen, P., and Malik, J. Recurrent network models for human dynamics. In ICCV,
2015. 1

Reed, S., Akata, Z., Mohan, S., Tenka, S., Schiele, B., and
Lee, H. Learning what and where to draw. In NIPS,
2016a. 2

Goroshin, R., Mathieu, M., and LeCun, Y. Learning to
linearize under uncertainty. In NIPS. 2015. 1

Reed, S., Akata, Z., Yan, X., Logeswaran, L., Schiele, B.,
and Lee, H. Generative adversarial text-to-image synthesis. In ICML. 2016b. 5

Ionescu, C., Papava, D., Olaru, V., and Sminchisescu, C.
Human3.6m: Large scale datasets and predictive methods
for 3d human sensing in natural environments. IEEE
Transactions on Pattern Analysis and Machine Intelligence, 36(7):1325–1339, jul 2014. 5, 7
Jayaraman, D. and Grauman, K. Learning image representations tied to ego-motion. In ICCV. 2015. 1
Jayaraman, D. and Grauman, K. Look-ahead before you
leap: end-to-end active recognition by forecasting the
effect of motion. arXiv preprint:1605.00164, 2016. 1
Krizhevsky, A., Sutskever, I., and Hinton, G. E. Imagenet
classification with deep convolutional neural networks.
In NIPS. 2012. 1, 5
Lee, N. Modeling of Dynamic Environments for Visual
Forecasting of American Football Plays. PhD thesis,
Carnegie Mellon University Pittsburgh, PA, 2015. 1
Lotter, W., Kreiman, G., and Cox, D. Deep predictive
coding networks for video prediction and unsupervised
learning. In ICLR. 2017. 1, 2
Mathieu, M., Couprie, C., and LeCun, Y. Deep multi-scale
video prediction beyond mean square error. In ICLR.
2016. 1, 2, 5
Michalski, V., Memisevic, R., and Konda, K. Modeling
deep temporal dependencies with recurrent "grammar
cells". In NIPS, 2014. 2
Mittelman, R., Kuipers, B., Savarese, S., and Lee, H. Structured recurrent temporal restricted boltzmann machines.
In ICML. 2014. 2

Shi, X., Chen, Z., Wang, H., Yeung, D.-Y., Wong, W.-k.,
and WOO, W.-c. Convolutional lstm network: A machine
learning approach for precipitation nowcasting. In NIPS.
2015. 5
Simonyan, K. and Zisserman, A. Two-stream convolutional
networks for action recognition in videos. In NIPS. 2014.
5
Simonyan, K. and Zisserman, A. Very deep convolutional
networks for large-scale image recognition. In ICLR,
2015. 6
Srivastava, N., Mansimov, E., and Salakhudinov, R. Unsupervised learning of video representations using lstms. In
ICML. 2015. 1, 2
Sutskever, I., Hinton, G. E., and Taylor, G. W. The recurrent
temporal restricted boltzmann machine. In NIPS. 2009. 2
Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S.,
Anguelov, D., Erhan, D., Vanhoucke, V., and Rabinovich,
A. Going deeper with convolutions. In CVPR. 2015. 1
Villegas, R., Yang, J., Hong, S., Lin, X., and Lee, H. Decomposing motion and content for natural video sequence
prediction. In ICLR. 2017. 1, 2, 7, 11
Vondrick, C., Pirsiavash, H., and Torralba, A. Generating
videos with scene dynamics. In NIPS. 2016. 2, 5
Walker, J., Gupta , A., and Hebert , M. Patch to the future:
Unsupervised visual prediction. In CVPR, 2014. 1

Learning to Generate Long-term Future via Hierarchical Prediction

Weiyu Zhang, M. Z. and Derpanis, K. From actemes to
action: A strongly-supervised representation for detailed
action understanding. In ICCV. 2013. 5
Xue, T., Wu, J., Bouman, K. L., and Freeman, W. T. Visual
dynamics: Probabilistic future frame synthesis via cross
convolutional networks. NIPS, 2016. 2
Yuen, J. and Torralba, A. A data-driven approach for event
prediction. In ECCV. 2010. 1

Learning to Generate Long-term Future via Hierarchical Prediction

Appendix
A. Motion-Based Pixel-Level Evaluation, Analysis, and Control Experiments
In this section, we evaluate the predictions by deciles of motion similar to Villegas et al. (2017) using Peak Signal-to-Noise
Ratio (PSNR) measure, where the 10th decile contains videos with the most overall motion. We add a modification to our
hierarchical method based on a simple heuristic by which we copy the background pixels from the last observed frame using
the predicted pose heat-maps as foreground/background masks (Ours BG). Additionally, we perform experiments based
on an oracle that provides our image generator the exact future pose trajectories (Ours GT-pose∗ ) and we also apply the
previously mentioned heuristics (Ours GT-pose BG∗ ). We put * marks to clarify that these are hypothetical methods as
they require ground-truth future pose trajectories.
In our method, the future frames are strictly dictated by the future structure. Therefore, the prediction based on the future
pose oracle sheds light on how much predicting a different future structure affects PSNR scores. (Note: many future
trajectories are possible given a single past trajectory.) Further, we show that our conditional image generator given the
perfect knowledge of the future pose trajectory (e.g., Ours GT-pose∗ ) produces high-quality video prediction that both
matches the ground-truth video closely and achieves much higher PNSRs. These results suggest that our hierarchical
approach is a step in the right direction towards solving the problem of long-term pixel-level video prediction.
A.1. Penn Action
In Figures 6, and 7, we show evaluation on each decile of motion. The plots show that our method outperforms the baselines
for long-term frame prediction. In addition, by using the future pose determined by the oracle as input to our conditional
image generator, our method can achieve even higher PSNR scores. We hypothesize that predicting future frames that
reflect similar action semantics as the ground-truth, but with possibly different pose trajectories, causes lower PSNR scores.
Figure 8 supports this hypothesis by showing that higher MSE in predicted pose tends to correspond to lower PSNR score.

20
18
16
14
12
11

21

26

31

Time Steps
8th decile

41

51

61

Ours
Ours BG
Ours GT-pose*
Ours GT-pose BG*
Conv LSTM
Optical Flow

24
22
20
18
16
14
1

11

21

31

Time Steps

41

51

Peak Signal to Noise Ratio

22

1

Peak Signal to Noise Ratio

9th decile
Ours
Ours BG
Ours GT-pose*
Ours GT-pose BG*
Conv LSTM
Optical Flow

61

Peak Signal to Noise Ratio

Peak Signal to Noise Ratio

10th decile

Ours
Ours BG
Ours GT-pose*
Ours GT-pose BG*
Conv LSTM
Optical Flow

24
22
20
18
16
14
12

1

11

21

31

Time Steps
7th decile

41

51

61

Ours
Ours BG
Ours GT-pose*
Ours GT-pose BG*
Conv LSTM
Optical Flow

26
24
22
20
18
16
14
12

1

11

21

31

Time Steps

41

Figure 6. Quantitative comparison on Penn Action separated by motion decile.

51

61

Learning to Generate Long-term Future via Hierarchical Prediction

22
20

Peak Signal to Noise Ratio

24

18
16
14
1

11

21

31

Time Steps
4th decile

41

28

51

24
22
20
18
16
14
11

21

31

Time Steps
2nd decile

41

51

61

26
24
22
20
18
16
14
1

11

21

31

Time Steps

41

22
20
18
16
14
12

51

61

11

21

31

Time Steps
3rd decile

41

51

61

Ours
Ours BG
Ours GT-pose*
Ours GT-pose BG*
Conv LSTM
Optical Flow

28
26
24
22
20
18
16
14
12
1

Ours
Ours BG
Ours GT-pose*
Ours GT-pose BG*
Conv LSTM
Optical Flow

28

24

1

Ours
Ours BG
Ours GT-pose*
Ours GT-pose BG*
Conv LSTM
Optical Flow

26

Ours
Ours BG
Ours GT-pose*
Ours GT-pose BG*
Conv LSTM
Optical Flow

26

61

Peak Signal to Noise Ratio

12

1

Peak Signal to Noise Ratio

5th decile
Ours
Ours BG
Ours GT-pose*
Ours GT-pose BG*
Conv LSTM
Optical Flow

Peak Signal to Noise Ratio

Peak Signal to Noise Ratio

Peak Signal to Noise Ratio

6th decile
26

11

21

31

Time Steps
1st decile

41

51

61

Ours
Ours BG
Ours GT-pose*
Ours GT-pose BG*
Conv LSTM
Optical Flow

32.5
30.0
27.5
25.0
22.5
20.0
17.5
15.0
1

11

21

31

Time Steps

41

51

61

Image Peak Signal to Noise Ratio

Figure 7. (Continued from Figure 6.) Quantitative comparison on Penn Action separated by motion decile.

22.5

1st decile
2nd decile
3rd decile
4th decile
5th decile
6th decile
7th decile
8th decile
9th decile
10th decile

20.0
17.5
15.0
12.5
10.0
7.5
5.0
2.5

0.02

0.04

0.06

0.08

0.10

Pose Mean Squared Error

0.12

0.14

Figure 8. Predicted frames PSNR vs. Mean Squared Error on the predicted pose for each motion decile in Penn Action.

The fact that PSNR can be low even if the predicted future is one of the many plausible futures suggest that PSNR may not
be the best way to evaluate long-term video prediction when only a single future trajectory is predicted. This issue might
be alleviated when a model can predict multiple possible future trajectories, but this investigation using our hierarchical
decomposition is left as future work. In Figures 9 and 10, we show videos where PSNR is low when a different future (from
the ground-truth) is predicted (left), and video where PSNR is high because the predicted future is close to the ground-true
future (right).

Learning to Generate Long-term Future via Hierarchical Prediction

t=17

t=40

t=54

t=60

Low PSNR

High PSNR

—————————————————————–

—————————————————————–

t=12

t=30

t=43

t=40

Low PSNR

High PSNR

Figure 9. Quantitative and visual comparison on Penn Action for selected time-steps for the action of baseball pitch (top) and
golf swing (bottom). Side by side video comparison can be found in our project website

Learning to Generate Long-term Future via Hierarchical Prediction

t=10

t=12

t=20

t=20

Low PSNR

High PSNR

—————————————————————–

—————————————————————–

t=5

t=25

t=11

t=40

Low PSNR

High PSNR

Figure 10. Quantitative and visual comparison on Penn Action for selected time-steps for the actions of jumping jacks (top) and
tennis forehand (bottom). Side by side video comparison can be found in our project website

Learning to Generate Long-term Future via Hierarchical Prediction

To directly compare our image generator using the predicted future pose (Ours) and the ground-truth future pose given by
the oracle (Ours GT-pose∗ ), we present qualitative experiments in Figure 11 and Figure 12. We can see that the both
predicted videos contain the action in the video. The oracle based video prediction reflects the exact future very well.

t=20

t=29

t=38

t=47

t=56

t=65

Groundtruth Ours GT-pose

Ours

Groundtruth Ours GT-pose

Ours

Groundtruth Ours GT-pose

Ours

t=11

Figure 11. Qualitative evaluation of our network for long-term pixel-level generation. We show the actions of baseball pitch (top
row), baseball swing (middle row), and gold swing (bottom row). Side by side video comparison can be found in our project
website.

Learning to Generate Long-term Future via Hierarchical Prediction

t=20

t=29

t=38

t=47

t=56

t=65

t=11

t=17

t=23

t=29

t=35

t=41

t=47

Groundtruth Ours GT-pose

Ours

Groundtruth Ours GT-pose

Ours

Groundtruth Ours GT-pose

Ours

t=11

Figure 12. Qualitative evaluation of our network for long-term pixel-level generation. We show the actions of tennis serve (top row),
clean and jerk (middle row), and tennis forehand (bottom row). We show a different timescale for tennis forehand
because the ground-truth action sequence does not reach time step 65. Side by side video comparison can be found in our project website.

Learning to Generate Long-term Future via Hierarchical Prediction

A.2. Human3.6M
In Figure 13, we show evaluation (PSNRs over time) of different methods on each decile of motion.

20
18
16
31

41

51

61

71

Time Steps
8th decile

81

24

91

101

111

Ours
Ours BG
Ours GT-pose*
Ours GT-pose BG*
Conv LSTM
Optical Flow

22
20
18
16
11

21

31

41

51

61

71

Time Steps
6th decile

81

91

101

111

24
22
20
18
16
1

11

21

31

41

51

61

71

Time Steps
4th decile

81

91

101

111

28
26
24
22
20
18
1

11

21

31

41

51

61

71

Time Steps
2nd decile

81

91

101

111

30
28
26
24
22
20
18
1

11

21

31

41

51

61

71

Time Steps

81

91

101

111

20
18
16

121

11

21

31

41

51

61

71

Time Steps
7th decile

81

91

101

111

121

91

101

111

121

91

101

111

121

Ours
Ours BG
Ours GT-pose*
Ours GT-pose BG*
Conv LSTM
Optical Flow

28
26
24
22
20
18
16
1

121

Ours
Ours BG
Ours GT-pose*
Ours GT-pose BG*
Conv LSTM
Optical Flow

32

Ours
Ours BG
Ours GT-pose*
Ours GT-pose BG*
Conv LSTM
Optical Flow

22

1

121

Ours
Ours BG
Ours GT-pose*
Ours GT-pose BG*
Conv LSTM
Optical Flow

30

24

121

Ours
Ours BG
Ours GT-pose*
Ours GT-pose BG*
Conv LSTM
Optical Flow

26

9th decile

26

121

Peak Signal to Noise Ratio

Peak Signal to Noise Ratio

21

26

1

Peak Signal to Noise Ratio

11

Peak Signal to Noise Ratio

Peak Signal to Noise Ratio

1

Peak Signal to Noise Ratio

Peak Signal to Noise Ratio

Ours
Ours BG
Ours GT-pose*
Ours GT-pose BG*
Conv LSTM
Optical Flow

Peak Signal to Noise Ratio

22

Peak Signal to Noise Ratio

Peak Signal to Noise Ratio

10th decile
24

11

21

31

41

51

61

71

Time Steps
5th decile

81

Ours
Ours BG
Ours GT-pose*
Ours GT-pose BG*
Conv LSTM
Optical Flow

30
28
26
24
22
20
18
16

1

11

21

31

41

51

61

71

Time Steps
3rd decile

81

Ours
Ours BG
Ours GT-pose*
Ours GT-pose BG*
Conv LSTM
Optical Flow

32
30
28
26
24
22
20
18

1

11

21

31

41

34

51

61

71

Time Steps
1st decile

81

91

101

111

121

91

101

111

121

Ours
Ours BG
Ours GT-pose*
Ours GT-pose BG*
Conv LSTM
Optical Flow

32
30
28
26
24
22
20
18

1

11

21

31

41

51

61

71

Time Steps

81

Figure 13. Quantitative comparison on Human3.6M separated by motion decile.

Learning to Generate Long-term Future via Hierarchical Prediction

Image Peak Signal to Noise Ratio

As shown in Figure 13, our hierarchical approach (e.g., Ours BG) tends to achieve PSNR performance that is better than
optical flow based method and comparable to convolutional LSTM. In addition, when using the oracle future pose predictor
as input to our image generator, the PSNR scores get a larger boost compared to Section A.1. This is because there is higher
uncertainty of the actions being performed in the Human 3.6M dataset compared to Penn Action dataset. Therefore, even
plausible future predictions can still deviate significantly from the ground-truth future trajectory, which can penalize PSNRs.
1st decile
2nd decile
3rd decile
4th decile
5th decile
6th decile
7th decile
8th decile
9th decile
10th decile

22
21
20
19
18
0.00

0.05

0.10

0.15

Pose Mean Squared Error

0.20

Figure 14. Predicted frames PSNR vs. Mean Squared Error on the predicted pose for each motion decile in Human3.6M.

To gain further insight on this problem, we provide two additional analysis. First, we compute how the average PSNR
changes as the future pose MSE increases in Figure 14. The figure clearly shows the negative correlation between the
predicted pose MSE and frame PSNR, meaning that larger deviation of the predicted future pose from the ground future
pose tend to cause lower PSNRs.
Second, we show snapshots of video prediction from different methods along with the PNSRs that change over time
(Figures 15 and 16). Our method tend to make plausible future pose trajectory but it can deviate from the ground-truth
future pose trajectory; in such case, our method tend to achieve low PSNRs. However, when the future pose prediction from
our method matches well with the ground-truth, the PSNR is much higher and the generated image frame is perceptually
very similar to the ground-truth frame. In contrast, optical flow and convolutional LSTM make prediction that often loses
the structure of the foreground (e.g., human) over time, and eventually their predicted videos tend to become static. It
is interesting to note that our method is comparable to convolutional LSTM in terms of PSNR, but that our method still
strongly outperforms convolutional LSTM in terms of human evaluation, as described in Section 6.2.
t=31

t=61

t=80

t=90

Low PSNR

High PSNR

Figure 15. Quantitative and visual comparison on Human 3.6M for selected time-steps for the action of walking (left) and walk
together (right). Side by side video comparison can be found in our project website.

Learning to Generate Long-term Future via Hierarchical Prediction

t=36

t=35

t=117

t=91

Low PSNR

High PSNR

—————————————————————–

—————————————————————–

t=48

t=61

t=93

t=109

Low PSNR

High PSNR

Figure 16. Quantitative and visual comparison on Human 3.6M for selected time-steps for the actions of walk dog (top left), phoning
(top right), sitting down (bottom left), and walk together (bottom right). Side by side video comparison can be found in our
project website.

Learning to Generate Long-term Future via Hierarchical Prediction

To directly compare our image generator using the predicted future pose (Ours) and the ground-truth future pose given by
the oracle (Ours GT-pose∗ ), we present qualitative experiments in Figure 17 and Figure 18. We can see that the both
predicted videos contain the action in the video. However, the oracle based video reflects the exact future very well.

t=29

t=47

t=65

t=83

t=101

t=119

Groundtruth Ours GT-pose

Ours

Groundtruth Ours GT-pose

Ours

Groundtruth Ours GT-pose

Ours

t=11

Figure 17. Qualitative evaluation of our network for long-term pixel-level generation. We show the actions of giving directions
(top three rows), posing (middle three rows), and walk dog (bottom three rows). Side by side video comparison can be found in our
project website.

Learning to Generate Long-term Future via Hierarchical Prediction

t=29

t=47

t=65

t=83

t=101

t=119

Groundtruth Ours GT-pose

Ours

Groundtruth Ours GT-pose

Ours

Groundtruth Ours GT-pose

Ours

t=11

Figure 18. Qualitative evaluation of our network for long-term pixel-level generation. We show the actions of walk together (top
three rows), sitting down (middle three rows), and walk dog (bottom three rows). Side by side video comparison can be found in
our project website.

