SynthCLIP: Are We Ready for a Fully Synthetic CLIP Training?
CLIP;synthetic data;generative
Yes
No ethics review needed.
- I suggest that the authors improve the introduction by explaining the motivations and use-cases of employing a purely synthetic dataset to train CLIP models.
- The paper provides a dataset of 30 million captioned images from diverse set of concepts.
- As the concepts are fixed, one can gather a subset of them to train new models with less concerns about NSFW contents leaking into the training data compared to using real images. 
- The models trained with similar dataset scales using synthetic captioned images show similar performance on down-stream tasks.
The paper explores the performance of CLIP-style models trained on purely synthetic image-caption pairs (called SynthCLIP) generated by modern text-to-image diffusion models and LLMs. It studies the scaling trends of such models and also provides a dataset of 30 million captioned images.
- The main weakness of the paper in my opinion is that the paper is not well-motivated. The introduction section does not provide convincing answers to the questions like "why should we use purely synthetic image-caption datasets? why not a hybrid approach? why is the problem significant?"

- Although controlling the concepts that are present in the dataset can be useful, there is no guarantee that the generated images for each concept are 1) faithful to the content and 2) do not contain NSFW content:
1) faithful to content: The described workflow only filters the captions, not the generated images. It is likely that the generated images contain noisy unrelated images. No workarounds in this regard has been proposed in the paper.

2) NSFW content: The latter may happen because the training of models like Stable diffusion have been on unfiltered datasets like LAION. It is possible that some NSFW contents appear with some concepts in these datasets frequently, resulting in generation of NSFW contents inadvertently. 

- Despite that the paper argues that one can use synthetic images from tail classes to augment the real datasets, I think it is not straightforward to do so. Although the idea seems sound, the Stable Diffusion (SD) model has been trained on real images that have the same long-tailed classes. Therefore, the performance of SD on these classes will not be satisfactory.
Yes
No ethics review needed.
1. I'm curious about the performance on ImageNet variants like ImageNetV2, ImageNet-A, ImageNet-R, and ObjectNet.

2. Will the performance of MLLM understanding tasks be enhanced by employing CLIP trained on SynthCI-30M?
1. The research detailed the pipeline for entirely synthetic creation of image-text pairs and introduced the dataset SynthCI-30M.

2. A comprehensive set of experiments was conducted to elucidate the efficiency and practical value of synthetic data, demonstrating its scalability and effectiveness.
It's interesting to see the emerging trend of training CLIP models using synthetic data. This work introduces SynthCLIP, a CLIP model training on synthetic data comprising both synthetic captions and images. The paper not only proposed the pipeline for creating synthetic data but also release SynthCI-30M, a comprehensive dataset housing 30 million captioned images generated entirely synthetically. This work unveiled the potential of leveraging synthetic data to enhance CLIP model training.
1. The zero-shot image classification results lack depth to gauge effectiveness comprehensively.

2. The experiments solely compared with CC3M and CC12M. How would results differ if a subset of LAION-400M is employed instead?
Yes
No ethics review needed.
1. It isn’t quite clear how MTL is calculated. What is it averaging over?
2. In an effort to understand differences in synthetic distributions versus natural distributions, how does performance change when using a CC3M concept distribution sample equalized to the real CC3M similar to Table 4? Another experiment to get at some of this would be taking CC3M, for each image prompting the model for a single “concept” then creating a caption and image using the proposed pipeline.
3. To understand the differences in scaling, it would be helpful to know the coefficients of the error with respect to dataset size on a log scale. How do natural and synthetic data coefficients compare?
4. This experiment is less pertinent than the above, but with results on improving the long tail distribution there might be interesting robustness properties as a result of concept representation. How does the real versus natural data compare on effective robustness in a framework like that of [1]?

Overall, the work is well presented but would benefit from a coverage of the first three points, and less importantly the fourth, to round out its presentation. I’d be happy to raise my score if the above are addressed.

[1] Nguyen et al. 2023 "Quality Not Quantity: On the Interaction between Dataset Design and Robustness of CLIP"
Each element of the synthetic data pipeline is soundly constructed, and follows community norms for composition. Furthermore, the work details a lot of the smaller design choices (e.g. LLM, prompt, and concept distribution) that contribute to the end to end system. The approach itself is noteworthy as it represents a full departure to synthetic data whereas most existing approaches require one of the modalities to be pre-existing.

Model benchmarks are extensive and representative of image-text model capabilities. They show both good diversity in dataset and task.

Some of the most interesting findings come from the paper's ablations. Figure 4a demonstrates the delta in performance that can be attributed to each different synthetic modality. In addition, it shows a potential failure mode of ignored generation commands and how they may be addressed. Additionally, the study on the concept bank is quite interesting providing support for the hypothesis that some of the difference in performance between natural and synthetic data is from the underlying conceptual distribution not a failure in quality. The experiments towards the mitigation of long-tail effects suggest an interesting direction for improving unseen or undersampled concepts in real world training.
The work proposes a synthetic training protocol for CLIP models extending prior work by creating both generated captions and generated images. In the process, the method demonstrates superior performance to common small scale image-text such as CC12M by curating a dataset of 30M synthetic examples. To better understand how different elements of the pipeline affect performance, they also ablate different choices of language models, differences caused by synthetic data sources, and how concept distribution impacts performance.
One concern with this work is that there is not sufficient evidence that the method might scale. Certain CLIP pretraining augmentations, like M3AE for example, have been shown to work at small scales, but yield no major benefit at larger scales [1]. Understanding that training frontier CLIP models is prohibitively expensive due to batch size needs, it’s sensible that this data is not available but worth keeping in mind.

The most immediate notice is the difference in data efficiency between real and synthetically drawn samples. Combined with the above, for practical applications it is not quite clear when one would adopt this method as it strictly leads to longer training runs and real training data is abundant (LAION-5B [2] and DataComp [3]). The method would benefit from further analysis into what is causing the reduction in performance, though some beginning analysis is done with respect to the concept distribution the gap is still left largely unexplained. 

[1] Weers et al. 2023 "Masked Autoencoding Does Not Help Natural Language Supervision at Scale"
[2] Schuhmann et al. 2022 "LAION-5B: An open large-scale dataset for training next generation image-text models"
[3] Gadre et al. 2023 "DataComp: In search of the next generation of multimodal datasets"
Yes
No ethics review needed.
- In lines 158-159 the authors state that the captioned are oriented for a single object. How would it effect the performance on tasks that require low-level details understanding? There are many works that try to train on detailed captions to incorporate such an understanding.

- TTI models are currently not good at generating text within images and in understanding relationships between objects. Wouldn't training on generated images result in a model with limited capabilities in such areas?

- Given figure 4, why would we need to generate images and not recaption ones that we can obtain easily by crawling the internet?
+ The paper is well written and easy to follow. 
+ Utilizing a purely synthetic dataset enables to control the data distribution and to collect data and any scale, without requiring human intervention.
+ Interesting and insightful ablation studies
The paper focuses on training a CLIP-based model using solely synthetic image-text pairs and study the effects of doing so.
SynthCLIP relies on the understanding that curating real multimodal data at scale comes with a cost of quality and alignment between images and their description. To this end, the authors propose to harness the advancement of Text-To-Image models and LLMs to generate a 30M purely synthetic dataset of image-caption pairs. Such a process enables to easily control the distribution of the data and to generate datasets at any scale with no human in the loop.
The authors study the effects of training CLIP with their propose dataset in a various of benchmarks including both vision and vision language tasks and compare it to training with real data.
- Empirical demonstration of the motivation - the authors claim that in real datasets, increasing their scale comes with a cost of quality and the proposed approach mitigates this and enables collecting quality data in any scale. However, empirically proving this requires comparing the performance against much larger real datasets than CC12M. Outperforming a model trained on CC12M with a 30M dataset doesn't showcase the advantage in quality of the synthetic data. Moreover, the model trained with a10M synthetic data has a worse performance compared to the model trained with a similar amount of real data. Thus, I am afraid that the main claim of the paper was not empirically demonstrated.

- Novelty - The proposed framework is based on existing models and approaches. Unfortunately, constructing the concept bank, which could have been an interesting place for novelty, is taken from an existing work.

- The necessity of "human intervention" - real datasets mainly scraped from the internet and the caption is achieved from the alt-text in the html. Thus, curating large datasets is done automatically. Indeed, such datasets are often noisy and there are various methods for filtering them (for example, by a CLIP-score threshold) or bootstrapping [1]. These methods enable collecting huge scale dataset without requiring human intervention. While relying on alt-text for caption often lead to short and oversimplified captions, there are many works that tackle this by proposing automatic recaptioning [2,3]. Thus, one can utilize real images oriented dataset with high quality without human intervention.

- Fairness of comparison in the experimental section - The authors have stated that the training is done for a fixed number of epochs with a fixed batch size for datasets of different size. From my understanding, this leads to a different number of training iterations for datasets at a different scale, impairing the validity of the comparison.

[1] Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation.
[2] Improving clip training with language rewrites.
[3] FuseCap: Leveraging Large Language Models for Enriched Fused Image Captions
We train CLIP on 30 million synthetic captions and images and draw insights.
@inproceedings{
anonymous2024synthclip,
title={Synth{CLIP}: Are We Ready for a Fully Synthetic {CLIP} Training?},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=7DY2Nk9snh},
note={under review}
}
We present SynthCLIP, a CLIP model trained on entirely synthetic text-image pairs. Leveraging recent text-to-image (TTI) networks and large language models (LLM), we generate synthetic datasets of images and corresponding captions at scale, with no human intervention. In this work, we provide an analysis on CLIP models trained on synthetic data. We provide insights on the data generation strategy, number of samples required, scaling trends, and resulting properties. We also introduce SynthCI-30M, a purely synthetic dataset comprising 30 million captioned images. Our code, trained models, and data, will be released upon acceptance.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
CLIP synthetic data generative
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/f1d6032386ea9df81be24e1d9a26313a3e2320d5.pdf
unsupervised, self-supervised, semi-supervised, and supervised representation learning
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
SynthCLIP: Are We Ready for a Fully Synthetic CLIP Training?
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission