Learning through Conditioning on Natural Language Feedback
Social Learning;Natural Language Feedback;Instructive Learning
In retrospect rushed and not ready for review, don't want to waste reviewers time
I have read and agree with the venue's withdrawal policy on behalf of myself and my co-authors.
We explore whether we can finetune language models by letting them generate answers conditioned on prior feedback.
@misc{
hillier2024learning,
title={Learning through Conditioning on Natural Language Feedback},
author={Dylan Hillier and Cheston Tan and Jing Jiang},
year={2024},
url={https://openreview.net/forum?id=2Sn0ty7zoI}
}
In this paper we explore the simple idea of teaching models by allowing them to condition their answers on natural language feedback. Motivated by the idea that natural language interactions provide a targeted, flexible, and level-appropriate reward signal, we study the ability of small instruction-tuned models to leverage feedback from a larger frontier model. We find while the frontier model provides generally high quality feedback, especially smaller models can struggle to use this due to noise in their generative output. After incorporating techniques like negative sampling, we find that models trained on these feedback-conditioned responses can perform similarly to those trained directly on teacher responses. We explore training using supervised finetuning and preference learning algorithms over a broad set of tasks including Big-Bench Hard. These findings are broadly applicable and our methods rely only on the ability of models to give and receive linguistic feedback. As such, they contribute to a growing body of work exploring how to best utilise the linguistic capabilities of language models for human-like instructive learning.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
~Dylan_Hillier1 ~Cheston_Tan1 ~Jing_Jiang1
Dylan Hillier Cheston Tan Jing Jiang
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Social Learning Natural Language Feedback Instructive Learning
I certify that there is no acknowledgement section in this submission for double blind review.
hillier|learning_through_conditioning_on_natural_language_feedback
/pdf/602938364839100b831b463126306b129a3e6944.pdf
foundation or frontier models, including LLMs
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
No
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
Learning through Conditioning on Natural Language Feedback
ICLR 2025 Conference Withdrawn Submission
ICLR.cc/2025/Conference/Withdrawn_Submission