Appendix
A Extended Related Works
We first review approaches for learning visual perception from static images before discussing strategies for learning
from video.
Weakly-Supervised Learning from Static Images
One family of approaches for learning visual perception from static images trains a visual encoder to predict the
representations of text captions often found accompanying images from the Web, as in CLIP (Radford et al., 2021) or
CoCa (Yu et al., 2022). The largest open source CLIP model to date, numbering 2B parameters and trained on over
2B web-scraped images (Cherti et al., 2023), demonstrates impressive performance on a wide range of downstream
image and video tasks. Notably, this is achieved using only the light-weight adaptation of task-specific heads, also
referred to as frozen-evaluation, and does not require expensive end-to-end fine-tuning of the pretrained model.
Self-Supervised Learning from Static Images
Other approaches for learning from static images leverage unsupervised objectives. Initial works on self-supervised
approaches are based on sparse coding or hand-crafted pretext tasks, such as colorization (Larsson et al., 2016, 2017),
rotation prediction (Gidaris et al., 2020), and jigsaws (Noroozi and Favaro, 2016). More recent approaches leverage
invariance-based objectives by training a visual encoder to be invariant to hand-crafted image transformations (Wu
et al., 2018; Chen et al., 2020).
Another family of methods learn representations using denoising autoencoders (Vincent et al., 2008); image inpainting
is one popular instantiation of this idea (Pathak et al., 2016). More recently, masked autoencoders (He et al., 2021)
train an encoder-decoder transformer to predict missing pixels of a masked image. Follow-up work addresses the
indeterminism of pixel reconstruction by exploring instantiations of masked image modeling in latent space (Baevski
et al., 2022b; Assran et al., 2023; Baevski et al., 2022a). These approaches can be seen as applications of the
predictive feature principle in the image modality.
There are also various methods that combine both masked image modeling and invariance criteria to learn visual
representations from static images, such as iBOT (Zhou et al., 2021) and DINOv2 (Zhou et al., 2021; Oquab et al.,
2023), the latter is currently the most competitive instantiation of self-supervised learning with static images, scaled
to a model with over 1.1B parameters trained on a curated dataset of 142M images.
Weakly-Supervised Learning from Videos
One family of approaches for learning visual perception from videos relies on weakly-supervised guidance from closed
captioning, often computed from an ASR transcription of audio data accompanying internet videos. For instance,
VideoBERT (Sun et al., 2019; Xu et al., 2021) trains a video encoder to predict masked spans in the textual closed
captions. Similarly, VideoCLIP (Xu et al., 2021) trains a video encoder to predict the representation of video
captions computed by a text encoder. Follow-up work such as MERLOT (Zellers et al., 2022), VATT (Akbari et al.,
2021), and InternVideo (Wang et al., 2022) extended VideoCLIP by incorporating additional unsupervised objectives.
Self-Supervised Learning from Videos
Similar to unsupervised learning from images, a family of unsupervised video representation learning approaches
enforces a spatio-temporal representation of a video clip to be invariant to hand-crafted spatio-temporal data
augmentations (Parthasarathy et al., 2022). However, one obvious insight is that the temporal ordering of visual
information in video can provide implicit supervision. Indeed, this insight is the key insight leveraged by many works
on unsupervised video learning. Towards leveraging temporal information as supervision, some approaches train a
visual encoder by predicting the temporal ordering of frames (Xu et al., 2019; Lee et al., 2017). Other approaches