Feature Prediction versus Pixel Reconstruction.
Approaches that predict in pixel space must dedicate
significant model capacity and compute to capture all
the low-level detail in the visual input. By contrast, ap-
proaches that predict in latent space have the flexibility
to eliminate irrelevant or unpredictable pixel-level details
from the target representation (Vondrick et al., 2016).
Predicting in representation space has been shown to
lead to versatile representations that perform well across
many downstream tasks through linear probing or low-
shot adaptation (Assran et al., 2023; Oquab et al., 2023;
Assran et al., 2022), while demonstrating an efficiency
gain during pretraining compared to pixel level recon-
struction (Assran et al., 2023; Baevski et al., 2022b,a).
The works of Baevski et al. (2022a,b) additionally show
that predicting in representation space results in compet-
itive end-to-end fine-tuning performance in the image,
audio and text domains. In this work, we extend these
findings to the video modality.
3 Methodology: Video-JEPA
x
x-encoder
predictorz
y
y-encoder
D(ˆsy, sy)
ˆsy
sy
Figure 2 Joint-Embedding Predictive Architectures are
trained to predict the representation of an inputy from
the representation of another inputx. The additional vari-
able z provides the predictor with information about the
transformation that computesy from x.
Our goal is to explore the effectiveness of feature pre-
diction as a stand-alone objective for learning visual
representations from video. To that end, we use a
joint-embedding predictive architecture (JEPA) (LeCun,
2022); see Figure 2. The main idea behind a JEPA is
to learn by predicting the representation of an inputy
from the representation of another inputx. The basic
architecture is made up of an encoder,Eθ (·), which com-
putes the representation of the inputs, and a predictor,
Pϕ (·), which predicts the representation ofy from the
representation ofx, conditioned on a variablez indicat-
ing the transformation (or corruption) betweenx and
y. Conditioning onz enables the generation of distinct
predictions for various transformations ofx.
3.1 Training Objective
We train our visual encoderEθ (·) to satisfy the con-
straint that representations computed from one part of
the video,y, should be predictable from representations
computed from another part of the video,x. The pre-
dictor networkPϕ (·), which maps the representation of
x to the representation ofy, is trained simultaneously
with the encoder, and is provided specification of the
spatio-temporal positions ofy through the conditioning
variable z ← ∆y .
Naively implementing the objective using the regression
minimizeθ,ϕ ∥Pϕ (Eθ (x), ∆y ) − Eθ (y)∥1,
would admit a trivial solution, where the encoder out-
puts a constant representation, regardless of its input.
In practice, we use the following modified objective to
prevent representation collapse,
minimizeθ,ϕ ∥Pϕ (Eθ (x), ∆y ) − sg(Eθ (y))∥1, (1)
where sg(·) denotes a stop-gradient operation, which
does not backpropagate through its argument, andEθ (·)
is an exponential moving average of the networkEθ (·).
The use of an exponential-moving average feature ex-
tractor along with a stop-gradient and a predictor has
been used as a collapse prevention strategy for image pre-
training (Grill et al., 2020), and studied empirically (Xie
et al., 2021) and theoretically (Tian et al., 2021). In
fact, the objective in equation(1) is similar to the loss
of Assran et al. (2023) used for image pretraining, but
we modify it to use anℓ1 regression, which we found to
be more stable.
Theoretical motivation. A theoretical motivation for
the effectiveness of this collapse prevention strategy was
proposed in Grill et al. (2020) for the BYOL method. We
provide a simple adaptation of their analysis for ourℓ1
loss. For ease of exposition, we will disregard the effect of
the conditioning variablez and consider one dimensional
representations. Denote the representation Eθ (y) by