Stacking Small Language Models for Generalizability
large language model;fine-tuning;natural language;efficient models;small language models
The submission is incomplete and cannot be reviewed without more details. The paper does not comply with the length requirements outlined in the CfP.
Submission Desk Rejected by Program Chairs
This paper evaluates stacks of fine-tuned small language models as a more cost-effective alternative to large language models.
@misc{
liang2024stacking,
title={Stacking Small Language Models for Generalizability},
author={Laurence Liang},
year={2024},
url={https://openreview.net/forum?id=Vn23PakSbM}
}
Recent advances show that large language models (LLMs) generalize strong performance across different natural language benchmarks. However, the large size of LLMs makes training and inference expensive and impractical to run in resource-limited settings. This paper introduces a new approach called fine-tuning stacks of language models (FSLM), which involves stacking small language models (SLM) as an alternative to LLMs. By fine-tuning each SLM to perform a specific task, this approach breaks down high level reasoning into multiple lower-level steps that specific SLMs are responsible for. As a result, FSLM allows for lower training and inference costs, and also improves model interpretability as each SLM communicates with the subsequent one through natural language. By evaluating FSLM on common natural language benchmarks, this paper highlights promising early results toward generalizable performance using FSLM as a cost-effective alternative to LLMs.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
~Laurence_Liang1
Laurence Liang
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
large language model fine-tuning natural language efficient models small language models
I certify that there is no acknowledgement section in this submission for double blind review.
liang|stacking_small_language_models_for_generalizability
/pdf/17c8e89fd9432a0264ac114aaa9fcf2a4b4a61e9.pdf
foundation or frontier models, including LLMs
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
Stacking Small Language Models for Generalizability
ICLR 2025 Conference Desk Rejected Submission
ICLR.cc/2025/Conference/Desk_Rejected_Submission