Two Stories in Understanding Arithmetic Learning at Scale
Math Arithmetic;Mechanistic Understanding;Large Language Models
We found comments that were unintentionally left uncommented in the main body of the text. Although this does not change our major conclusions and insights, it may violate the double-blind policy of ICLR. Therefore, we have decided to withdraw this version and we apologize for any inconvenience this may cause.
I have read and agree with the venue's withdrawal policy on behalf of myself and my co-authors.
We find that language models perform arithmetic learning in a pure symbolic way, and we further provide a holistic framework to understand how.
@misc{
deng2024two,
title={Two Stories in Understanding Arithmetic Learning at Scale},
author={Chunyuan Deng and Zhiqi Li and Roy Xie and Ruidi Chang and Hanjie Chen},
year={2024},
url={https://openreview.net/forum?id=i7aKFeGf1D}
}
Large Language Models (LLMs) have long been suspected of struggling with arithmetic learning due to the inherent differences between language modeling and numerical calculation, but concrete evidence for this claim has been lacking. In this work, we respond to this claim through a two-side experiment. First, we investigate whether LLMs leverage partial products during arithmetic learning. We find that although LLMs can identify some partial products after learning, they fail to leverage them during the learning process. Instead, these partial products appear to be by-products from the pattern fitting. Second, we examine whether LLMs treat arithmetic in a purely symbolic manner. We decompose the task into subgroup-level (pairs of subportion tokens), hypothesizing that arithmetic learning difficulty arises from subgroup complexity, subgroup selection, and sequential prediction. We find that, given controlled subgroup complexity, LLMs treat a collection of different arithmetic operations similarly. Furthermore, arithmetic with low entropy in the subgroup label space lends to be more learnable. By analyzing position-level accuracy across different training sizes, we further explore subgroup selection. We observe that position-level accuracy follows a U-shaped pattern: LLMs quickly learn the easiest patterns at the first and last positions, while progressively learning the more difficult patterns in the middle positions. This phenomenon suggests that LLMs follow an easy-to-hard learning mechanism based on subgroup complexity. Our work confirms that LLMs are pure symbolic learners in arithmetic tasks and underscores the importance of understanding them deeply through subgroup-level quantification.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
~Chunyuan_Deng1 ~Zhiqi_Li4 ~Roy_Xie1 ~Ruidi_Chang1 ~Hanjie_Chen1
Chunyuan Deng Zhiqi Li Roy Xie Ruidi Chang Hanjie Chen
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Math Arithmetic Mechanistic Understanding Large Language Models
I certify that there is no acknowledgement section in this submission for double blind review.
deng|two_stories_in_understanding_arithmetic_learning_at_scale
other topics in machine learning (i.e., none of the above)
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
Two Stories in Understanding Arithmetic Learning at Scale
ICLR 2025 Conference Withdrawn Submission
ICLR.cc/2025/Conference/Withdrawn_Submission