text: Weight-decay is also linearly increased from 0.04 to 0.4 throughout pretraining. The y-encoder weights are initialized 
  identically to the x-encoder, and subsequently updated as an exponential moving average (EMA) (Tarvainen and 
  Valpola, 2017) of the x-encoder weights using a momentum value which starts at 0.998 and is linearly increased to 
  1.0 during training (Caron et al., 2021; Assran et al., 2022). We scale all hyper-parameter schedules 25% beyond 
  the actual training schedule. Specifically, the learning rate schedule, weight-decay schedule, and EMA schedule 
  are computed assuming a training length of 112,500 iterations, even though we only train our model for 90,000 
  iterations. We found the last 25% of the default scheduler period to update hyper-parameters too aggressively, and 
  simply truncating the schedulers improved performance. 
  
  Masking. As described in Section 3, we propose a 3D Multi-Block masking strategy. We use two type of masks: 
  short-range masks, where we take the union of 8 randomly sampled target blocks with a spatial scale of 0.15, and 
  long-range masks, where we take the union of 2 randomly sampled target blocks with a spatial scale of 0.7. In both 
  cases, the aspect ratio for all sampled blocks is randomly chosen in the range (0.75, 1.5). 