Table 3 Average Pooling vs. Adaptive Pooling.We pool the
feature map output by the frozen V-JEPA encoder using
an attentive probe, which is then fed into a linear classifier
for downstream supervised tasks (K400 and SSv2). We
evaluate two pooling strategies: 1) average pooling (Avg.),
and attentive pooling (Att.). Results are reported using
a single center view. Using adaptive pooling with a cross-
attention layer leads to improvements of+17.3 points on
K400 and+16.1 points on SSv2.
Frozen Evaluation
K400 SSv2
(16×1×1) (16 ×1×1)
Method Arch. Avg. Att. Avg. Att.
V-JEPA ViT-L/16 56.7 73.7 50.1 66.2
has been critical for enabling the surge of advancements
in other modalities, such as text and images (Kaplan
et al., 2020; Cherti et al., 2023). We investigate whether
a similar trend holds for video data. To control for the
possible confounding variable of compute budget, we
pretrain all models in Table 2 for 90K iterations using
a batch-size of 3072. We report downstream results on
K400, SSv2, and IN1K using a frozen backbone with an
attentive probe, and report top-1 accuracy using a single
center view.
Table 2 shows that average performance across tasks
monotonically increases as we increase the size of the
pretraining dataset, but the best task-specific perfor-
mance is obtained by independently selecting the pre-
training data for each specific downstream task. For
instance, the L/16 obtains its best SSv2 performance
when pretrained on K710+SSv2, its best K400 perfor-
mance when pretrained only on K710, and its best IN1K
performance when pretrained only on K710+HT. The
best average performance across all tasks is achieved by
pretraining VideoMix2M, which combines all the data
sources. Similarly, the H/16 pretrained on K710+SSv2
achieves a greater K400 score than the H/16 pretrained
on VideoMix2M, however, the top performing H/16 on
average is pretrained on VideoMix2M.
4.3 Evaluation: Attentive Probing
Next we explore the feature pooling strategy for apply-
ing the model’s representations in downstream tasks.
Since the prediction objective in equation(1) is unnor-
malized, there is no a priori reason for the encoder to
yield a linearly separable subspace (Chen et al., 2020).
Thus, rather than using a linear operation (averaging)
to pool the features output of the frozen backbone, we
explore a learnable non-linear pooling strategy. Specifi-
cally, when evaluating the frozen pretrained backbone
on downstream tasks, we learn a cross-attention layer
with a learnable query token. The output of the cross-
attention layer is then added back to the query token
(residual connection), and then fed into two-layer MLP
Table 4 Ablating Prediction Task. Models are ViT-L/16
networks pretrained on K710 and SSv2 and evaluated with
an attentive probe using a single center view. The regionx is
sampled by masking spatio-temporal regions in the video;y is
the mask complement.1) random-tube[r]:x is obtained by
masking a fractionr of tubes (spatial patches extended across
the entire temporal duration) from the video,2) causal
multi-block[p]: x is restricted to the firstp frames of the
16-frame video, which are then masked with a random set
of spatio-temporal blocks,3) multi-block: x is obtained
by masking a random set of spatio-temporal blocks from the
entire video. Best performance obtained by using multiblock
masking.
Frozen Evaluation
K400 SSv2 IN1K
Masking (16×1×1) (16 ×1×1)
random-tube[0.9] 51.5 46.4 55.6
causal multi-block[6] 61.3 49.8 66.9
causal multi-block[12] 71.9 63.6 72.2
multi-block 72.9 67.4 72.8
with a single GeLU activation, followed by a LayerNorm,
and finally a linear classifier.
In Table 3 we see that using adaptive pooling with
a learnable cross-attention layer leads to a significant
improvement of+17 points on K400 and+16.1 points
on SSv2. Using an attentive-probe is also beneficial for
other baseline models as reported in Appendix E.
4.4 Prediction Task: Predictingy from x
We conduct an ablation on the masking strategy used in