Measuring Bias of Web-filtered Text Datasets and Bias Propagation Through Training
LLMs;text datasets;classification;bias;rewrite;propagation
Yes
No ethics review needed.
see above
1. The study provides a detailed look at biases in seven widely used LLM pretraining datasets. Revealed biases persist even when text is rephrased by other LLMs. 

2. By showing that dataset biases are measurable, persistent, and propagate into LLM-generated outputs. It suggests that even datasets created with strict filtering and deduplication standards still exhibit biases, emphasizing the need for new methods to mitigate these issues.
This paper investigates the biases present in LLM pretraining datasets and examines how these biases persist and propagate through training. The study calms different datasets possess unique biases or fingerprints identifiable by models, even when preprocessed similarly or rewritten. Shows that classifiers can distinguish dataset origin with high accuracy, and biases can carry over.
1. My main concern is the study use of prompt-based rephrasing to test bias persistence introduces potential confounding effects, as prompts may inadvertently impose their own linguistic patterns or styles. This prompt influence could create artifacts that the classifier detects, rather than the underlying biases in the original datasets.

2. The study sticks mostly to a 160M model, barely looking into how bigger models, like the billion-parameter ones used in real applications, might handle and spread dataset biases.  Without testing scalability, it’s unclear if the study’s conclusions hold for bigger, more powerful models where bias effects could still remain or reduce.
Yes
No ethics review needed.
Refer to weaknesses.
- The research question is interesting and impactful to the LLM research direction.
- The authors conduct extensive experiments, such as the impact of text rewrite,  and draw their findings in a rigorous way.
This work investigates the distinguishability of a range of popular open-source pretraining text datasets derived from CommonCrawl, including C4, RefinedWeb, DolmaCC, RedPajama-V2, FineWeb, DCLMBaseline, and others. The study presents interesting findings: 1) a classifier trained on these datasets achieves high accuracy on held-out test data, despite humans finding the task challenging; 2) this distinguishability extends to models pre-trained on each dataset—specifically, a classifier trained on the original text datasets performs well in distinguishing between models pre-trained on these datasets.
- It would be insightful to further investigate whether distinguishability propagates to models fine-tuned on the same downstream task. For instance, if models are pre-trained on different text datasets but fine-tuned on the same dataset, will their behaviors remain distinguishable?
- Considering that the construction of the pre-training datasets involves only data filtration, without any modification or augmentation, and that these datasets share similar sources, it seems counterintuitive that they are distinguishable at the level of individual segments. Could the authors provide further explanation on this?
- The study claims the existence of dataset bias by demonstrating corpus distinguishability. It would be beneficial to identify and describe more explicit dimensions of bias, as this would offer clearer implications and impact.
Yes
No ethics review needed.
1. How do you ensure the classification accuracy on generated text isn't simply detecting general "AI-generated text" patterns rather than dataset-specific biases?

2. Have you tested if these biases persist through fine-tuning or RLHF? This seems crucial given current LLM development practices.
1. The paper shows how different filtering pipelines create distinct "fingerprints" in the data, even when using similar preprocessing steps.

2. The paper does comprehensive ablation studies examining key factors like model size, training data amount, and sequence length. These controlled experiments help isolate the important variables affecting classification accuracy. The validation approach using multiple methods (human studies, rewriting experiments, bias propagation tests) strengthens the findings by showing the robustness of the results across different experimental paradigms.
This paper examines biases in popular pretraining datasets for large language models, demonstrating that transformer models can distinguish between texts from different datasets (like C4, RefinedWeb, and DolmaCC) with surprisingly high accuracy, despite these datasets being derived from CommonCrawl using similar filtering methods. Through user studies and rewriting experiments, the authors show these biases are subtle to humans but persistent through reformatting, and importantly, they propagate through training - models trained on these datasets inherit their distinctive characteristics. The work includes comprehensive ablation studies and extends similar dataset bias research from computer vision.
1. It doesn't deeply analyze what features enable this classification. A feature importance analysis (e.g., using attention weights or gradient-based attribution methods) could reveal which textual patterns or structures the classifier relies on, providing actionable insights for dataset creators.

2. The rewriting experiments use only GPT-4 for text modification. Testing with multiple different LLMs would strengthen the finding that biases persist through rewriting. Additionally, more controlled rewriting experiments (e.g., systematically modifying specific text features like sentence length, vocabulary complexity, or discourse markers) could better isolate which characteristics contribute to dataset fingerprints.

3. While the paper demonstrates dataset biases exist and propagate, it doesn't propose concrete methods to mitigate them.
Yes
No ethics review needed.
1. Regarding the classification accuracy, it is unclear whether these authors choose these dataset combinations for classification experiments. I suggest the authors do a 2-way classification with respect to each dataset pairs, leading to a  matrix or heatmap showing the 2-way classification accuracies between all dataset pairs. This would give a clearer picture of which datasets are most/least distinguishable from each other.
2. The conclusion of dataset bias is valid, but could the authors do more investigation on the critical differences that differentiate between different datasets ? For example, changing some paraphrases in Category 1 may alter the classifier results to Category 2, thus these paraphrases may be a bias in Category 1. I could understand that it is hard to enumerate over all data samples, but some interpretable examples will be appreciated, such as a few concrete examples of text that are particularly indicative of each dataset. 
3. The last section seems to be a draft without comprehensive evaluation. Some details are not clear, for example, the prompt for sentence generation from these LMs, different prompts on the impact of the classification accuracy. I would suggest a more structured evaluation framework for this section, such as a comparison of results across different models or datasets.
1. The idea is interesting and the problem of data bias in large language models pre-training dataset is an important challenge in the community
2. The proposed evaluation method is solid and reasonable/
This method proposes an interesting method to measure bias of web-filtered text datasets, and evaluate the bias propagation through training the large language models. The idea is insightful and the experiments are in general solid, but there are still several concerns to be addressed.
1. The evaluation part needs to be more thorough, questions are listed in the next section
2. The architecture of the paper could be better organized for improving readability.
@inproceedings{
anonymous2024measuring,
title={Measuring Bias of Web-filtered Text Datasets and Bias Propagation Through Training},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=FDhAngvHuf},
note={under review}
}
In this paper, we investigate biases in pretraining datasets for large language models (LLMs) through dataset classification experiments. Building on prior work demonstrating the existence of biases in popular computer vision datasets, we analyze popular open-source pretraining text datasets derived from CommonCrawl including C4, RefinedWeb, DolmaCC, RedPajama-V2, FineWeb, DCLM-Baseline, and others. Despite those datasets being obtained with similar filtering and deduplication steps, LLMs can classify surprisingly well which dataset a single text sequence belongs to, significantly better than a human can. This indicates that popular pretraining datasets have their own unique biases or fingerprints. Those biases remain even when the text is rewritten with LLMs. We also demonstrate that these biases propagate through training: Random sequences generated by models trained on those datasets can be classified well by a classifier trained on the original datasets.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
LLMs text datasets classification bias rewrite propagation
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/e7d9b39bd72c0337c8f58d2013932bea9713a375.pdf
foundation or frontier models, including LLMs
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
/attachment/23048e1e701c91ecb7de37fbb9bdcfc56cddb8a3.zip
Measuring Bias of Web-filtered Text Datasets and Bias Propagation Through Training
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission