To that end, we pretrain a family ofV-JEPA models
on a dataset of 2 million videos collected from pub-
licly available datasets by combining a masked modeling
prediction task with a joint-embedding predictive ar-
chitecture (see Figure 2). We measure performance on
several downstream image and video tasks, using both
frozen evaluation and end-to-end fine-tuning. Our find-
ings suggest that feature prediction can indeed serve as
an effective stand-alone objective for unsupervised learn-
ing from video, while using significantly shorter training
schedules than pixel prediction methods. Specifically:
• Feature prediction leads to versatile visual repre-
sentations that perform well across downstream
image and video tasks without adaption of the
model’s weights; i.e., using a frozen backbone.
V-JEPA achieves the best performance among
methods we consider (+6% accuracy) on the
SomethingSomething-v2 task, which requires fine-
grained temporal understanding. V-JEPA is
also competitive on tasks like Kinetics400, where
appearance-based features are sufficient and hence
state-of-the-art image models such as DINOv2 excel
(Figure 1 and Table 6).
• Models trained with feature prediction are supe-
rior to pixel prediction approaches under a frozen
evaluation protocol (attentive probing) and are com-
petitive with pixel prediction under full fine-tuning,
while using significantly shorter training schedules
(Tables 5 and 6).
• Models trained with feature prediction are more
label-efficient than pixel prediction approaches. De-
creasing the available number of labeled examples re-
sults in an increase in the performance gap between
V-JEPA and pixel-reconstruction models (Table 7).
2 Related Works
Slow Features. One way to encourage temporally
adjacent representations to be predictive of each other
is to ensure that they vary slowly over time. Early
works targeting predictive features encouraged represen-
tations of individual video frames to be locally tempo-
rally invariant, while preventing representation collapse
by using spectral methods, as in SFA (Wiskott and Se-
jnowski, 2002), SSA (Kayser et al., 2001), and Simulated
Fixations (Zou et al., 2012). More recently, Goroshin
et al. (2015); Wang et al. (2010) train a siamese con-
volutional network to map the representations of two
subsequent frames to the same point, while encouraging
distant frames to have diverse representations via a pair-
wise margin loss and a triplet loss, respectively. Other
works (Oord et al., 2018; Surís et al., 2021; Feichtenhofer
et al., 2021) implement temporal invariance using noise-
contrastive estimation (Gutmann and Hyvärinen, 2012).
Our exploration in this paper goes beyond temporal in-
variance and explores feature prediction using masked
modeling.
Predictive Features. Going beyond local invariance,
a family of works trains a predictor network to map the
representation of a frame or clip at one time-step to a
distinct representation at another time-step. Srivastava
et al. (2015); Vondrick et al. (2016); Wang et al. (2023b)
train such a video feature predictor network on top of
a frozen pretrained image or video encoder. Unfreezing
the target feature extractor, several methods train the
video encoder and the predictor network simultaneously,
while preventing collapse by using a supervised action
forecasting loss (Girdhar and Grauman, 2021), or by
using the representations of distant clips as negative
samples in a contrastive loss (Han et al., 2019, 2020;
Tan et al., 2023), often focusing on small convolutional
encoders (Han et al., 2019, 2020). The idea of learning a
representation by predicting missing information in fea-
ture space is also core to the joint-embedding predictive
architecture (JEPA) (LeCun, 2022), which combines a
siamese encoder with a predictor network. JEPAs have
been successfully instantiated in several modalities, such
as with audio data (Baevski et al., 2022b) and image
data (Zhou et al., 2021; Oquab et al., 2023; Assran et al.,