2019), Kinetics-400/600/700 (K710) (Kay et al., 2017),
and Something-Something-v2 (SSv2) (Goyal et al., 2017),
and remove any overlap with the validation sets of
Kinetics-400/600/700 and Something-Something-v2, re-
sulting in approximately 2 million videos. We train a
ViT-L/16, a ViT-H/16, and a ViT-H/16384 transformer
model on VideoMix2M. We use a batch size of 3072 for
the ViT-L/16 and ViT-H/16 models, and a batch size
of 2400 for the ViT-H/16384 model. Each model takes
as input a video clip of 16 frames sampled with a frame-
skip of 4, corresponding to roughly 3 second clips on
average. The ViT-L/16 and ViT-H/16 process the video
at a spatial resolution of 224, while the ViT-H/16384
uses an input resolution of 384; cf. Appendix C.
4