MoDE: CLIP Data Experts via Clustering
Jiawei Ma Po-Yao Huang Saining Xie Shang-Wen Li Luke Zettlemoyer Shih-Fu Chang Wen-Tau Yih Hu Xu







MoDE: CLIP Data Experts via Clustering
    Jiawei Ma^*,   Po-Yao Huang^   Saining Xie^3   Shang-Wen Li^ 

Luke Zettlemoyer^,4   Shih-Fu Chang^   Wen-Tau Yih^   Hu Xu^+ 

^1FAIR, Meta   ^2Columbia University   ^3New York University   ^4University of Washington


    25 April 2024
==============================================================================================================================================================================================================================

^* Research done while Jiawei Ma was an intern at FAIR.
^+ Project Lead.



The success of contrastive language-image pretraining (CLIP) relies on the supervision from the pairing between images and captions, which tends to be noisy in web-crawled data. We present Mixture of Data Experts (MoDE) and learn a system of CLIP data experts via clustering. Each data expert is trained on one data cluster, being less sensitive to false negative noises in other clusters. At inference time, we ensemble their outputs by applying weights determined through the correlation between task metadata and cluster conditions. 
To estimate the correlation precisely, the samples in one cluster should be semantically similar, but the number of data experts should still be reasonable for training and inference. As such, we consider the ontology in human language and propose to use fine-grained cluster centers to represent each data expert at a coarse-grained level. 
Experimental studies show that four CLIP data experts on ViT-B/16 outperform the ViT-L/14 by OpenAI CLIP and OpenCLIP on zero-shot image classification but with less (<35%) training cost. Meanwhile, MoDE can train all data expert asynchronously and can flexibly include new data experts. The code is available at <https://github.com/facebookresearch/MetaCLIP/tree/main/mode>.





sections_arxiv/introduction
sections_arxiv/relatedwork
sections_arxiv/approach

sections_arxiv/experiment
sections_arxiv/discussion



§ CONCLUSION


The success of CLIP depends on the quality negative samples. As the false negative noise in web-crawled pairs hurts training effectiveness, scaling CLIP on large-scale data presents unique challenges in terms of training efficiency and computational bottlenecks.
To this end, we have presented Mixture of Data Experts () to asynchronously train a group of data experts. Each expert model is trained on a set of fine-grained clusters where the data in each cluster is of coherent semantics and all data experts are trained individually.
During inference, the outputs are selectively ensembled based on the requirements for each task and modeled by the correlation between task metadata and fine-grained cluster centers.
Empirically,  significantly outperforms  OpenCLIP and OpenAI CLIP on standard benchmarks with less than 35% training cost.
Furthermore, the image embeddings extracted by all data experts can be combined to enhance the representation of visual information.
We plan to adapt MoDE for generative models in the future.




§ ACKNOWLEDGEMENT

The authors would like to thank Xinlei Chen and Margaret Li for constructive discussion.



ieee_fullname



sections_arxiv/appendix

