Architectures that include text as one of their modalities can gain a large assortment of common-sense knowledge from Internet data, which leads to better generalization in low-data modalities. We introduce Vision Oui, an image classification architecture capable of harnessing text-only datasets in addition to text-image pairs. We train on a filtered subset of the Common Crawl corpus before finetuning on the LAION-5B dataset. The resulting model achieves SOTA on 13 different benchmarks with comparable performance to industry standard models. We further distill the model for 3x performance gains while retaining 99% accuracy.
spam