Revisiting Feature Prediction for Learning Visual
Representations from Video
Adrien Bardes1,2,3, Quentin Garrido1,4, Jean Ponce3,5,6, Xinlei Chen1, Michael Rabbat1, Yann LeCun1,5,6,
Mahmoud Assran1,†, Nicolas Ballas1,†
1FAIR at Meta,2Inria, 3École normale supérieure, CNRS, PSL Research University,4Univ. Gustave Eiffel,
CNRS, LIGM,5Courant Institute, New York University,6Center for Data Science, New York University
†Joint last author
This paper explores feature prediction as a stand-alone objective for unsupervised learning from video and
introduces V-JEPA, a collection of vision models trained solely using a feature prediction objective, without
the use of pretrained image encoders, text, negative examples, reconstruction, or other sources of supervision.
The models are trained on 2 million videos collected from public datasets and are evaluated on downstream
image and video tasks. Our results show that learning by predicting video features leads to versatile visual
representations that perform well on both motion and appearance-based tasks, without adaption of the
model’s parameters; e.g., using a frozen backbone. Our largest model, aViT-H/16 trained only on videos,
obtains 81.9% on Kinetics-400,72.2% on Something-Something-v2, and77.9% on ImageNet1K.
Date: April 15, 2024
Correspondence: {abardes, massran, ballasn}@meta.com
Code: https://github.com/facebookresearch/jepa
Blogpost: Click here
1 Introduction
Humans possess the remarkable ability to map low-level
signals originating from the retina into a semantic spatio-
temporal understanding of the world; synthesizing no-
tions such as objects and global motion (Spelke et al.,
1995). A long-standing goal of the machine learning
community is to identify the principles or objectives that
may guide such unsupervised learning in humans (Field,
1994; Berkes and Wiskott, 2005; Hinton, 1989). One
related hypothesis is based on the predictive feature
principle (Rao and Ballard, 1999), which posits that
representations of temporally adjacent sensory stimuli
should be predictive of each other.
In this work, we revisit feature prediction as a stand-
alone objective for unsupervised learning of visual repre-
sentations from video. Numerous advances in the field —
such as the standard use of transformer architectures in
vision (Dosovitskiy et al., 2020), the maturing of masked
autoencoding frameworks (Xie et al., 2021; Bao et al.,
2021; He et al., 2021), query-based feature pooling (Chen
et al., 2022), joint-embedding predictive architectures
(JEPA) (LeCun, 2022; Assran et al., 2023; Baevski et al.,
2022b), and larger datasets — form a unique arsenal of
tools, which we integrate in a modern and conceptually
simple method, thevideo joint-embedding predictive ar-
chitecture or V-JEPA, which is based solely on feature
prediction, without using pretrained image encoders,
text, negative examples, human annotations, or pixel-
70 72 74 76 78 80 82 84 86 88 90 92
40
50
60
70
SOTA fine-tuned task-specific
model on SSv 2 (MVD)
SOTA fine-tuned
task-specific model
on K 400 (UniFormer)
ViT-L/16
V-JEPA
ViT-H/16
DINOv2
ViT-g/14
OpenCLIP
ViT-G/14
I-JEPA
ViT-H/16
Hiera
Hiera-H
VideoMAE
ViT-H/16
VideoMAEv2
ViT-g/14
OmniMAE
ViT-H/16
Kinetics 400
Something-Something-v2
Frozen Evaluation
Video Feature Pred.
Video Pixel Pred.
Image Models
Figure 1 V-JEPA models pretrained on video learn versatile
visual representations. It performs well on motion-based
tasks (Something-Something-v2) and appearance-based tasks
(Kinetics 400) without adaptation of the model’s parameters,
i.e., using the same frozen backbone for both tasks.
level reconstruction.
We seek to answer the simple question:
How effective is feature prediction as a stand-
alone objective for unsupervised learning from
video with modern tools?
1
arXiv:2404.08471v1  [cs.CV]  15 Feb 2024