text: has been critical for enabling the surge of advancements 
  in other modalities, such as text and images (Kaplan 
  et al., 2020; Cherti et al., 2023). We investigate whether 
  a similar trend holds for video data. To control for the 
  possible confounding variable of compute budget, we 
  pretrain all models in Table 2 for 90K iterations using 
  a batch-size of 3072. We report downstream results on 
  K400, SSv2, and IN1K using a frozen backbone with an 
  attentive probe, and report top-1 accuracy using a single 
  center view. 
  
  with a single GeLU activation, followed by a LayerNorm, 
  and finally a linear classifier. 
  
  In Table 3 we see that using adaptive pooling with 
  a learnable cross-attention layer leads to a significant 
  improvement of +17 points on K400 and +16.1 points 
  on SSv2. Using an attentive-probe is also beneficial for 
  other baseline models as reported in Appendix E. 
  
  Table 2 shows that average performance across tasks 
  monotonically increases as we increase the size of the 
  pretraining dataset, but the best task-specific perfor- 
  mance is obtained by independently selecting the pre- 
  training data for each specific downstream task. For 
  instance, the L/16 obtains its best SSv2 performance 
  when pretrained on K710+SSv2, its best K400 perfor- 
  mance when pretrained only on K710, and its best IN1K 
  performance when pretrained only on K710+HT. The 
  best average performance across all tasks is achieved by 
  pretraining VideoMix2M, which combines all the data 
  sources. Similarly, the H/16 pretrained on K710+SSv2 
  achieves a greater K400 score than the H/16 pretrained 
  on VideoMix2M, however, the top performing H/16 on 
  average is pretrained on VideoMix2M. 