Frozen
x-encoder
 predictor
 decoder
(a) Visualization Methodology.We train a conditional diffusion model to decode the V-JEPA feature-space predictions to
interpretable pixels; the pretrained V-JEPA encoder and predictor networks are kept frozen in this process. The decoder is
only fed the representations predicted for the missing regions of the video, and does not have access to the unmasked regions
of the video.
(b) Visualizations. First Row:Masked videos used as input to the V-JEPA models (a pretrained ViT-H/16 encoder and its
corresponding predictor network).Other rows:Bounding boxes contain various samples from the decoder overlayed on the
original video. V-JEPA is not a generative model and the decoder does not have access to the context (first row), so we do
not expect samples to exactly match the input. This experiment qualitatively illustrates what information is encoded and
predicted by V-JEPA. In particular, characteristics that are common across samples represent information that is encoded in
the V-JEPA predictions. V-JEPA generates predictions that are spatially and temporally coherent with unmask region of the
video. The predictions also capture consistent motion through time.
Figure 6 Qualitative Analysis. Offline visualizations of the V-JEPA feature-space predictions.
7 Conclusion
In this work, we explored the effectiveness of feature
prediction as a stand-alone objective for unsupervised
learning from video and introducedV-JEPA, a collection
of vision models trained solely using a self-supervised
feature prediction objective. TheV-JEPAmodels demon-
strate the ability to solve various downstream image and
video tasks without adaption of the model parameters,
and outperform previous video representation learning
approaches in frozen evaluation on action recognition,
spatio-temporal action detection, and image classifica-
tion tasks. Additionally, we show that pretrainingV-
JEPA on videos is particularly effective for solving down-
stream tasks requiring fine-grained motion understand-
ing, while large-scale image models trained on internet
scale datasets fall short on such tasks. Finally, we em-
pirically observed thatV-JEPA models are label-efficient
learners, and exhibit good performance on downstream
tasks, even when only few labeled examples are available.
References
Hassan Akbari, Liangzhe Yuan, Rui Qian, Wei-Hong Chuang,
Shih-Fu Chang, Yin Cui, and Boqing Gong. Vatt: Trans-
formers for multimodal self-supervised learning from raw
video, audio and text.Advances in Neural Information
Processing Systems, 34:24206–24221, 2021.
10