text: Optimization. We use AdamW (Loshchilov and Hutter, 2017) to optimize the x-encoder and predictor weights. 
  The ViT-L/16224 and ViT-H/16224 models use a batch size of 3072 while the ViT-H/16384 uses a batch size of 
  2400. Models are trained for a total of 90,000 iterations. The learning rate is linearly increased from 2 
  10−4 
  10−4 during the first 12, 000 iterations of pretraining, and decayed to 10−6 following a cosine schedule. 
  to 6.25 
  × 
  × 
  
  17 