text: Pretraining. We combine several public datasets to 
  construct an unsupervised video pretraining dataset, 
  which we refer to as VideoMix2M. Specifically, we com- 
  bine the videos from HowTo100M (HT) (Miech et al., 
  2019), Kinetics-400/600/700 (K710) (Kay et al., 2017), 
  and Something-Something-v2 (SSv2) (Goyal et al., 2017), 
  and remove any overlap with the validation sets of 
  Kinetics-400/600/700 and Something-Something-v2, re- 
  sulting in approximately 2 million videos. We train a 
  ViT-L/16, a ViT-H/16, and a ViT-H/16384 transformer 
  model on VideoMix2M. We use a batch size of 3072 for 
  the ViT-L/16 and ViT-H/16 models, and a batch size 
  of 2400 for the ViT-H/16384 model. Each model takes 
  as input a video clip of 16 frames sampled with a frame- 
  skip of 4, corresponding to roughly 3 second clips on 
  average. The ViT-L/16 and ViT-H/16 process the video 
  at a spatial resolution of 224, while the ViT-H/16384 
  uses an input resolution of 384; cf. Appendix C. 