Studying the Effects of Training Data on Small Language Models
small language models;pretraining
Yes
No ethics review needed.
See Weaknesses.

Typos:
1. line 19: propeties -> properties
2. line 86: exihibit -> exhibit
3. line 527: thire -> their
1. The paper tested a very meaningful assumption of whether simple language in training data can lead to better generation abilities of SLMs.
2. The readability measurement approaches are comprehensively studied and analyzed.
The authors investigates the impact of training data's readability to the generation abilities of very small language models (SLM). They challenge the claim that training SLMs on simple language is the reason for their ability to generate coherent text. They create synthetic corpora with varying level of readability, and found no impact to the coherence of text generated by SLMs, and also found training on simple language does not lead to earlier development of coherence during training.
1. The quality measurement is limited to perplexity and coherence (coherent, according to the llm-as-judge prompt, is considered "well-structured and well-organized", "not just a heap of related information, but should build from sentence to sentence"). The ignorance of other dimensions of quality (for example, as authors also mentioned, clarity and fluency) makes any statements about "generation abilities of SLMs" an overclaim.
2. The quality measurement doesn't use any metrics from the original TinyStories paper: grammar, creativity, consistency with the beginning of the story (Eldan & Li, 2023). That makes the results from the two papers in comparable. Because of that, there is no evidence that "SLMs trained on data with substantially more complex language also exhibit the same abilities as those trained on simple language" can also hold the measurement in Eldan & Li (2023).
3. While the authors rule out some factors not contributing to coherent SLMs, it is unclear what factors are contributing.
Yes
No ethics review needed.
I would love to hear the authors' response to my interpretable of the tables / figures, in case there is any misunderstanding.

I am open to raising my score if there is a strong argument for why correcting this misunderstanding is important for the community, as it is my main concern about the paper.
- The paper is cleanly scoped and clearly written.
- It corrects a widespread misinterpretation of a result in the NLP literature. This result has been used to motivate LM development inspired by human language learning.
This paper investigates the question: are small LMs capable of learning TinyStories because it is *readable* (i.e., simple vocabulary, concepts, and grammatical structures) or some other feature, notably the dataset's lack of diversity (templated sentences with uniform structure)? The authors of TinyStories, and subsequent citations, only consider the former interpretation, but there is no evidence to eliminate the latter.

This paper carefully investigates this question by generating two datasets with the same synthetic data generation process, differing only in the vocabulary and the intended audience that the model is asked to use & consider. They call these two datasets $\\texttt{LlamaTales-Jr}$ and $\\texttt{LlamaTales-GRE}$. The two datasets are equally coherent, but $\\texttt{LlamaTales-Jr}$ is much more readable. They find that small LMs are *equally* capable of learning both $\\texttt{LlamaTales-Jr}$ and $\\texttt{LlamaTales-GRE}$, showing that *readability* does not necessarily explain small LMs' ability to learn TinyStories. Instead, they hypothesize it is the lack of diversity in the data.
- The scope of the paper is relatively narrow. While it shows that the community has widely misinterpreted the results of a particular paper, it's not clear how much it matters. Moreover, I believe the main surprising finding of $\\texttt{TinyStories}$ still stands, which is that SLMs are capable of learning the language of 3-4 year olds (regardless of why).
- I believe the overall paper can use some reorganization.
    - I find it odd that §3 and §4 (which are all about measuring the readability and quality of the existing dataset, $\\texttt{TinyStories}$) are ordered before §5 (about constructing the datasets used in this paper). Wouldn't it make more sense to first describe the data creation methodology, *then* validate that they have the expected readability and quality? Right not, we don't get to the meat of the paper until halfway through page 7.
    - The connection between figures and claims in the running text of the paper is all over the place. For instance, most of the main claims in §3 and §4 are supported by figures in the Appendix.
- The presentation of tables and figures can be more readable.
   - Figures 2, 3, 6 are hard to interpret due to lack of textual explanation, and I think there must be a better way to present the results. My understanding is that in Figure 2, I should see that in (b), the *green* dots (SLMs trained on $\\texttt{LlamaTales-Jr}$) are approximately as high as the best gray dots (LLMs), and in (c), the *blue* dots (SLMs trained on $\\texttt{LlamaTales-GRE}$) are ALSO approximately as high as the best gray dots (LLMs). Wouldn't it be better for these to be on the same axes, so the reader can compare directly whether LlamaTales-GRE is as learnable as LlamaTales-Jr? Subplots (a) for $\\texttt{TinyStories}$ and (d) for $\\texttt{FineWeb}$ should be in the Appendix, since they aren't used to support the main claims. I'm not sure what Figure 3 is doing in the main paper, since it's not discussed in the running text.
    - Table 1 contains results for many metrics which are not discussed in the running text of the main paper. To prevent reader confusion, I recommend moving the results for these metrics to the Appendix, where the metrics are described. The different metrics also don't seem to tell a different story.
    - I recommend a table with examples from $\\texttt{LlamaTales-Jr}$ and $\\texttt{LlamaTales-GRE}$.
Yes
No ethics review needed.
When measuring quality, you use a set of open models. Why not simply use a state of the art model such as GPT-4o or Claude3.5 instead?
The paper provides a new dataset with some added features in relation to a previous dataset (TinyStories).
This paper discusses the effects on training small language models by changing some features related to the concept of readability of one particular dataset. The experiments show no effects.
The scientific contribution of this paper is limited, as it tackles a very narrow (and somewhat artificial) research question, and the experiments show no discernible effects whatsoever. The research question is somewhat artificial in the sense that the concept of readability (in humans) concerns the cognitive load of *interpreting* a text, which is not the same thing as *learning* a statistical language model from a text. In particular since readability is usually defined in terms of features related to frequency and length of individual tokens, but the paper does not discuss the influence of tokenization on the learning abilities of language models. It is therefore not at all clear (to me) why the concept of readability would have anything at all to do with how well a statistical language model performs. The experiments included in the paper confirms that it does not. The paper also contains an experiment that shows that the concept of readability and the concept of text quality (as interpreted in terms of perplexity and coherence) are unrelated, which is exactly what you would expect given the definition of these concepts. As such, it is difficult to see what novel knowledge this paper contributes with.
@inproceedings{
anonymous2024studying,
title={Studying the Effects of Training Data on Small Language Models},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=4xBew7kuYB},
note={under review}
}
Prior work has found that training very small language models (SLMs) on synthetic children's stories allows them to generate coherent text, comparable to much larger models. These stories are claimed to encompass the vocabulary and factual knowledge base of a 3-4 year old child, capturing the "essence of natural language."
Because of these claims, it is tempting to attribute the findings to the simple language of children's stories, drawing a parallel to how children learn language.
Is the human concept of readability relevant in the context of language model training, or are these findings better explained by other propeties of the data?
In this study, we investigate this by first validating several automatic readability measures. We then create synthetic corpora with varying levels of readability and assess the coherence of text generated by SLMs trained on these corpora.
We find no relationship between the readability of training data and the generation abilities of SLMs. Specifically, SLMs trained on data with substantially more complex language also exihibit the same abilities as those trained on simple language. Moreover, training on simple language does not lead to earlier development of coherence during training.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
small language models pretraining
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/407060a3df878a3a13c95e416ede391a6d5cd987.pdf
other topics in machine learning (i.e., none of the above)
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
Studying the Effects of Training Data on Small Language Models
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission