been successfully instantiated in several modalities, such
as with audio data (Baevski et al., 2022b) and image
data (Zhou et al., 2021; Oquab et al., 2023; Assran et al.,
2023). In this work, we extend this paradigm to video
data by leveraging recent advances in self-supervised
learning.
Advances in Self-Supervised Learning.The use
of vision transformers (Dosovitskiy et al., 2020; Li et al.,
2022) has become standard practice in self-supervised
learning with joint-embedding architectures (Chen et al.,
2021; Caron et al., 2021; Oquab et al., 2023; Zhou et al.,
2021; Assran et al., 2022), and unlocked masked image
modeling in pixel space by parameterizing the pixel de-
coder as a transformer with learnable mask tokens (Doso-
vitskiy et al., 2020; Xie et al., 2021; He et al., 2021; Bao
et al., 2021), demonstrating a step-change in the rep-
resentation quality of autoencoding methods (Vincent
et al., 2010). This line of generative methods was sub-
sequently extended to video data using spatio-temporal
masking (Tong et al., 2022; Feichtenhofer et al., 2022;
Wang et al., 2023a; Kalluri et al., 2023; Gupta et al.,
2023). It was also recently shown that the representa-
tionsofmaskedimageautoencoderscouldbesignificantly
improved by using learnable pooling mechanisms based
on cross-attention (Chen et al., 2022). Finally, through
careful selection of design choices, the non-contrastive
collapse prevention strategy in BYOL (Grill et al., 2020)
was recently made to work with image feature prediction
methods (Baevski et al., 2022b; Assran et al., 2023),
which demonstrated the ability to learn representations
that can be leveraged for various downstream tasks with-
out relying on invariance to hand-crafted image trans-
formations.
2