B Extended Description of V-JEPA
In this section, we provide an in-depth description of our approachV-JEPA that is illustrated in Figure 3.
Input. Unless stated otherwise, during during pretraining, we always randomly sample a clip of 16 frames from
each input video with a temporal stride of 4 between sampled frames. An input video clip therefore covers 64 frames
in total, or roughly 2 seconds of a given video running at 30 frames per second. We then resize the video’s spatial
dimensions to224 × 224, resulting in an overall shape of16 × 224 × 224 × 3 for the entire clip. Since ViT networks
process a 1D sequence of tokens, we must convert an input video clip into a 1D token sequence. To do so, we apply a
3D convolution comprisingd filters of size2 × 16 × 16 with a temporal stride of2 and a spatial stride of16, resulting
in a tensor of shape8 × 14 × 14 × d. Next we add absolute 3D sin-cos positional embeddings to the spatio-temporal
feature map and flatten it, resulting in a 1D token sequence of shape1568 × d. This process is demonstrated in
Figure 7.
[16 x 224 x 224 x 3]
3D Conv
[2 x 16 x 16 x d]
[8 x 14 x 14 x d]
3D sin-cos absolute position 
embeddings
[8 x 14 x 14 x d]
[1568 x d]
+16 video frames
resolution 224 x 224
flatten
Figure 7 V-JEPA training operates on a video clip flattened into a sequence of tokens. To convert a video clip of size
16 ×224 ×224 ×3 into a 1D token sequence, we apply a 3D convolution comprisingd filters of size2 ×16 ×16 with a temporal
stride of2 and a spatial stride of16, resulting in a tensor of shape8 ×14 ×14 ×d. Next we add absolute 3D sin-cos positional
embeddings to the spatio-temporal feature map and flatten it, resulting in a 1D token sequence of shape1568 × d.
V-JEPA. We sample both a video clip, and a video mask in each iteration. We denote a video clip represented as
a 1D token sequence of lengthL = 1568 by xL = (x1, . . . , xL). Similarly, given a mask ofM < Lpatches, leaving
N = L − M patches unmasked, we denote the indices of masked patches by(i1, . . . , iM ) and its complement (the
indices of unmasked patches) by(j1, . . . , jN ).
Computing thex-representations. To compute theV-JEPA loss, we first produce thex-representations by masking
the video clip and feeding it into thex-encoder; we denote the masked video byxN = (xj1 , . . . , xjN ). Applying thex-
encoder Eθ(·) to the masked clip gives a sequence of patch representations, denoted aszN = Eθ(xN ) = (zj1 , . . . , zjN ).
Predicting the target. Next, the V-JEPA predictor network Pϕ(·, ·) takes as input the tokens produced by the
x-encoder and predicts the missing regions in the video clip, which are specified by a set of learnable mask tokens.
Specifically, the mask tokens are parameterized as the sum of a shared learnable vector and an absolute 3D
sin-cos positional embedding, denoted bymM = (mi1 , . . . , miM ). The output of the predictor is thus given by,
ˆsM = Pϕ(zN , mM ) = (ˆsi1 , . . . ,ˆsiM ), corresponding to ad-dimensional output for each of theM masked patches.
Computing the y-representations. Finally to compute the prediction targets, the entire unmasked video clip is
processed by they-encoder to obtain a set of target representations, denoted bysL = Eθ(xL) = (s1, . . . , sL). The
V-JEPA loss is now computed as
Loss = 1
M
X
k∈(i1,...,iM)
∥ˆsk − sk∥1, (2)
which is simply the averageL1 distance between the output of the predictor and they-encoder. We then compute a
gradient update with respect to the parameters of thex-encoder, θ, and the predictor,ϕ, and subsequently update
the parameters of they-encoder as an exponential moving average of the context encoder weights (Polyak average).
16