[L×d]
 [N×d]
\
Remove
masked
tokens
Binary Mask
[T×H×W]
Eθ
x-encoder
[N×d]
 [L×d]
Concatenate
mask tokens
Pφ
predictor
[M×d]
 [M×d]
 [L×d]
/
Remove
unmasked
tokens
E ¯θ
y-encoder
[L×d]
L1 / /
stop-grad
Figure 3 V-JEPA. Training operates on a video clip ofT frames with spatial resolutionH × W, flattened into a sequence
of L tokens. (Left to right): We first obtain the input of thex-encoder by dropping tokens from the video clip. The
x-encoder then processes the masked video sequence, and outputs an embedding vector for each input token. Next, the
outputs of thex-encoder are concatenated with a set of learnable mask tokens containing positional embeddings of the masked
spatio-temporal patches. The predictor network processes the combined token sequence, and outputs an embedding vector for
each mask token. The outputs of the predictor are then regressed to the prediction targets using anL1 loss. The prediction
targets correspond to the output of they-encoder.
3.2 Prediction Task: Predictingy from x
The feature prediction task is based on a masked mod-
eling formulation (He et al., 2021; Tong et al., 2022);
i.e., regionsx and y from the video are sampled using
masking. To sampley from a video, we sample several
(possibly overlapping) spatially continuous blocks with
various aspect ratios and repeat the spatial blocks across
the entire temporal dimension of the video;x is taken to
be the complement. Masking a large continuous block
that covers the full temporal dimension limits informa-
tion leakage due to the spatial and temporal redundancy
of videos, and results in a harder prediction task (Tong
et al., 2022).
We leverage two types of masks: short-range masks,
where we take the union of8 randomly sampled target
blocks covering 15% of each frame, and long-range masks,
where we take the union of2 randomly sampled target
blocks covering 70% of each frame. In both cases, the
aspect ratio for all sampled blocks is randomly chosen in
the range(0.75, 1.5). Given that both short-range and
long-range masks are produced by sampling many blocks
and taking their union, the result is an average masking
ratio of ∼ 90%. We refer to our masking strategy as
multi-block, and compare it to other possible masking
strategies in Section 4.
3.3 Network Parameterization
We use a Vision Transformer (ViT) (Dosovitskiy et al.,
2020; Arnab et al., 2021) as our video backbone. To
process a video with a transformer network, we split the
video clip into a 3D grid ofL spatio-temporal patches,
where a patch consists of a16 × 16 pixel block spanning
2 consecutive frames; we refer to these spatio-temporal
patches as tokens. This sequence of tokens is then di-
rectly processed by the stack of transformer blocks. In-
puts x and y correspond to masked regions of a video, we
apply the video masks by simply dropping a subset of the
tokens. We apply masking at the input of thex-encoder,
and at the output of they-encoder to construct contex-
tualized targets (Baevski et al., 2022b). The encoder is
parameterized using standard ViT networks, while the
predictor is a narrow transformer implemented using
12 blocks with an embedding dimension of384. Taking
inspiration from masked autoencoders (He et al., 2021),
our predictor takes as input the sequence of embeddings
produced by thex-encoder as well as a sequence of learn-
able mask tokens with positional embeddings indicating
the spatio-temporal positions of they tokens. The out-
put of the predictor is an embedding vector for each
mask token; see Figure 3 and refer to Appendix B for
more details.
3.4 Pretraining Data and Evaluation Setup
Pretraining. We combine several public datasets to
construct an unsupervised video pretraining dataset,
which we refer to as VideoMix2M. Specifically, we com-
bine the videos from HowTo100M (HT) (Miech et al.,
2019), Kinetics-400/600/700 (K710) (Kay et al., 2017),
and Something-Something-v2 (SSv2) (Goyal et al., 2017),
and remove any overlap with the validation sets of