text: Advances in Self-Supervised Learning. The use 
  of vision transformers (Dosovitskiy et al., 2020; Li et al., 
  2022) has become standard practice in self-supervised 
  learning with joint-embedding architectures (Chen et al., 
  2021; Caron et al., 2021; Oquab et al., 2023; Zhou et al., 
  2021; Assran et al., 2022), and unlocked masked image 
  modeling in pixel space by parameterizing the pixel de- 
  coder as a transformer with learnable mask tokens (Doso- 
  vitskiy et al., 2020; Xie et al., 2021; He et al., 2021; Bao 
  et al., 2021), demonstrating a step-change in the rep- 
  resentation quality of autoencoding methods (Vincent 
  et al., 2010). This line of generative methods was sub- 
  sequently extended to video data using spatio-temporal 
  masking (Tong et al., 2022; Feichtenhofer et al., 2022; 
  Wang et al., 2023a; Kalluri et al., 2023; Gupta et al., 
  2023). It was also recently shown that the representa- 
  tions of masked image autoencoders could be significantly 
  improved by using learnable pooling mechanisms based 
  on cross-attention (Chen et al., 2022). Finally, through 
  careful selection of design choices, the non-contrastive 
  collapse prevention strategy in BYOL (Grill et al., 2020) 
  was recently made to work with image feature prediction 
  methods (Baevski et al., 2022b; Assran et al., 2023), 
  which demonstrated the ability to learn representations 
  that can be leveraged for various downstream tasks with- 
  out relying on invariance to hand-crafted image trans- 
  formations. 