The following table displays the different model configurations we experimented with. We also determine the maximum batch size that was able to fit in memory for each configuration in the experiments section.



Approx Model params
Hidden size
MLP size
Heads
Unimodal layers
Multimodal layers
Model size (fp32)




350M (original)
768
3072
12
12
6
1.33GB


900M
1024
4096
16
24
12
3.48GB


1.8B
1280
5120
16
32
16
6.66GB


2.7B
1408
6144
16
40
20
10.3GB


