Rethinking Memorization in LLMs: On Learning by Rote vs. with Understanding
language models;memorization;generalization
Yes
No ethics review needed.
To trigger forgetting, a more realistic experiment would be to train a model on a dataset where some examples are specifically marked (with a prefix hash-key). And then check if the model can exactly produce those samples, when prompted with the hash key. Then perform sequential fine-tuning on a new distribution, and see if the model has forgotten to produce previously learnt samples. Crucially, in this setting, make sure that the size of the training set is big (> 10000 samples). Would you still expect to see the same behavior as in Section-4?
- This work looks at the challenging problem of identifying the interplay between generalization and memorization in LLMs, and designs some interesting experiments to study this.
- One interesting experiment is looking at what properties of the data cause memorization, where the authors find increasing entropy of the data distribution increases the model’s propensity to memorize, while lower entropy causes models to enter into the generalization mode.
The paper looks at fine-tuning LLMs (Pythia, LLama, Mistral) on strings generated from PCFGs. They analyze the resulting training dynamics and find a generalization phase, and then a memorization phase. These phases are different for different LLMs, and also differ based on whether the training data came from a PCFG or is randomly sampled. Note however that the size of the training data is unusually small and always less than 64 strings.

Next, the paper tries to understand the interplay between memorization and generalization by fine-tuning sequentially on 2 samples $D_1$ and $D_2$,  drawn from the same distribution. They find that training on $D_1$ first causes loss on the test set to go down, and then go up with extended training. After training on $D_1$, training on $D_2$ causes loss to increase on $D_1$, while loss on the test set starts going down again. The authors conclude that they can trigger forgetting of previously memorized examples by memorizing new data from the same distribution, which further can also trigger a round of generalization (since the test set loss also starts going down).
- I think the biggest weakness for the paper is that it re-frames many existing and well-known observations from standard machine learning. For instance the memorization / generalization plots are simply the U-shaped val loss plots seen in ML 101 classes. This is commonly referred to as over-fitting. I’m not sure if there’s any deeper insights to be gathered from just that one plot (which the paper heavily relies on).
- Similarly, the fact that loss will increase on a small train set of < 32 examples, after over-fitting to it and then starting to train on a second training sample, is also extremely well-known. 
- The size of the datasets used here is unusually small, which makes me wonder if many of the conclusions are an artifact of the small data size.
Yes
No ethics review needed.
* does the syntax support testing hierarchical/compositional generalization?

* line 98: "memorization begins at epoch 6": modern LLM didn't go through data for more than 1 epoch sometimes, how would the author explain the big difference here?

* Line 317: what does it mean by "takes a very long time"? decoding time?

* Line 370-371 "one must generate test and trainign data...": I can't imagine scenario like this. What do the authors mean by "generate"?

* how would the author comment if the defined memorization term (Line 356-358) happens to be negative?
* the experiments consider various aspect of the generalization & memorization aspect.

* the new way to quantize memorization feels like a progress. And a revisiting of the traditional bias-variance trade-off seems to be interesting in that it makes the finer-grain distinction that there could be a phase of purely understanding before the traditional trade-off point ("end of learning by understanding"). 

* Line 418 - 420: this feels like an interesting observation.
The authors study the factor of memorization (rote) and generalization (understanding) during training. The experiments are conducted on training and testing data on synthetic data generated from context-free grammar. The authors propose a new way to quantify memorization, and find that rote learning harms understanding and by developing understanding.
* missing citation and discussion. A more comprehensive study of generalization [2, 3] exists; and CFG data was used for conducting thorough experiments [1] before, and thus is not novel (but the author made them sound novel in abstract Line 16). Also, the second experiment (Figure 4) is closely connected to continual learning, but no discussion of relevant previous results are made.

* some of important point is not addressed in the paper.

Line 73-74: "start of learning by rote" --- Why is learning previous to this not by rote? "end of learning by understanding" --- When is the start of learning by understanding?

* the training set sizes and test sizes are too small. Despite this, the observation is still a bit confusing: train and test loss has small gap, while the author give the impression in Line 196-200 that the output space is large. I think this might be caused by simple CFG grammar. Would be nice to provide some insight for why.

* I disagree with some of the interpretation of results. Or the use of terms by the authors are not rigorous.

Line 422-424: First I think catastrophic forgetting happens in continual learning setting. I don't think the model forgets entirely ("destroy") about D_train,1 as the loss is still lower than D_test.

Line 426 "destroy the ability to generalize...": what's the observation for "destroy"?

Line 428-429 "differences between human and machine cognition": I think this spectulation is taken to far from what's in the paper.

Line 499-500 "Intuitively, .... to memorize": the phrasing seems in-accurate. It's harder to generalize with higher entropy distribution, so the model is **encouraged** to be for the model to be a lookup table. I don't know why it is "easier".

Line 521-522 "measures of memorization must ...": Why is it a "must"? 



[1]: Physics of Language Models: Part 1, Learning Hierarchical Language Structures
[2]: COGS: A Compositional Generalization Challenge Based on Semantic Interpretation
[3]: SLOG: A Structural Generalization Benchmark for Semantic Parsing
Yes
No ethics review needed.
Does "training size n=8, n=64" refer to the number of items in the training dataset?
1.  The topic under investigation is meaningful.
This paper attempts to explore the learning patterns of large language models, specifically whether they learn by rote or by understanding. The authors generate synthesis data via probabilistic context-free grammar(PCFG) as training data and compare loss curves of different LLMs such as Pythia and LLaMA. The main conclusion is that by observing the curves between training loss and test loss, it can infer the learning patterns of the language models. If the training loss continues to decrease while the test loss increases, it is considered to be learning by rote.
1. The phenomenon of learning by rote discussed in this paper is similar to overfitting. However, their difference is never explained. If a small model of 128M is pre-trained on WikiText-103 for lots of epochs, a similar phenomenon occurs as well. Essentially, it is overfitting—when the model's capacity exceeds the information provided by the text.  It lacks some quantitative experimental data and rigorous argumentation. For example, the relationship between the amount of information in the pre-train corpus and the model parameters required to alleviate rote memorization, and how to estimate the amount of information. Thus current findings are difficult to constitute actual contributions. The conclusion also lacks practical significance. When the test loss diverges from the training loss, it indicates learning by rote, which is just another name for overfitting.

2. The synthetic data used is too trivial and lacks any distinguishing properties like other works(https://arxiv.org/abs/2305.13673). It simply generates strings based on PCFG, without any extra knowledge or inner logic. To determine whether the model is based on understanding or rote memorization, there needs to be at least some semantic-level properties. For example, executable formal languages. LLMs can understand complex logic, such as sorting and human-defined functions.  For example, making the model fit the results of executable formal languages may help us determine whether the model has achieved true understanding rather than learning by rote.
Yes
No ethics review needed.
How was the PCFG designed?
The paper finds that:
- memorization usually begins before generalization ends
- continued training on a different training set restarts generalization and causes forgetting of the first training set
- lower-entropy distributions are easier to generalize on; higher-entropy distributions are more amenable to memorization
This paper studies learning curves of various language models on some synthetic languages (generated from a unigram model and a nonrecursive PCFG). They define memorization, or learning by rote, to begin at the point that a gap between the training and test loss begins (more precisely, when training loss > 1.05 * test loss), and generalization, or learning by understanding, to end when test loss is minimized.

-
The paper sets for itself the goal of _how_ models memorize (l. 159), but to me this sounds like studying the mechanisms internal to the model that enable it to memorize, whereas this study is a study of learning curves only.

The terms of "learning by rote" and "learning with understanding" are central to the paper. The experiments all center around identifying the beginnings and ends of these two kinds of learning, and one of the aims of the paper is to propose these two definitions as a better alternative to the tests for memorization in previous work. Given the centrality of these two concepts, I would like to see (1) clearer definitions of them earlier in the paper (they are defined only briefly at lines 98-99, and more clearly not until page 6) and (2) some justification for why your definitions of these terms are the right ones.

I'd suggest that you use a single term for each concept, instead of using "learning by rote" and "memorization" as synonyms and "learning with understanding" and "generalization" as synonyms.

The experiments are on synthetic languages instead of naturally occurring data. The reason is to be able to control things like the vocabulary size or the entropy. But there's nothing about the experiments that depends on any details of the data distribution, and the findings are basically the same for the unigram model and the PCFG.

In the end, the findings are not very surprising. I feel that the insights that can be gained by looking at learning curves alone are limited.
We investigate the interplay between learning by rote (memorization) and learning with understanding in large language models using formal grammars.
@inproceedings{
anonymous2024rethinking,
title={Rethinking Memorization in {LLM}s: On Learning by Rote vs. with Understanding},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=hFQZmKFtlT},
note={under review}
}
Understanding whether and to what extent token sequences generated by large language models (LLMs) are the result of regurgitating memorized training data or are based on meaningful learning of the training data's syntax and semantics has many important implications.
In order to cleanly measure and disentangle token recollection by rote (memorization) from generation with understanding, we create an experimental framework that is based on training LLMs over *sequences generated using formal grammars*. Our framework allows us to better understand the interplay between the two types of learning, namely, *by rote* vs. *with understanding*. Using our framework we make several striking observations that hold consistently across different open-source model families (Pythia, Llama, and Mistral): (a) we find that the learning types are at odds with each other during training, i.e., rote learning harms understanding and by developing understanding, models forget previously memorized sequences, (b) we find that *entropy of the training datasets* impacts the ease of learning, with lower entropy datasets being easier to learn with understanding and higher entropy datasets being easier to learn by rote, (c) we highlight the difficulty of determining the type of learning involved in a model based solely on recollecting a training data sequence. Our surprising results have significant downstream implications in the study and usage of LLMs.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
language models memorization generalization
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/d05de401028ac8b43f4ead695be62a0ec9bde865.pdf
foundation or frontier models, including LLMs
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
/attachment/002d094480e47e261f547da35af14d69ec6a48f9.zip
Rethinking Memorization in LLMs: On Learning by Rote vs. with Understanding
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission