Is In-Context Learning Sufficient for Instruction Following in LLMs?
Large Language Models;In-context Learning;Alignment
Yes
No ethics review needed.
- The abbreviation URAIL is not defined.
- Line 139: If providing multi-turn examples improves performance at multi-turn conversation, why do the authors not study it in this paper?
- The results in Figure 2 are not representative in my opinion since the confidence bounds are overlapped largely. 
- Line 320: You say including examples from SkillMix is better than including examples from URIAL. Does URIAL come with a curated dataset as well? 
- Figure 3: I don't get it clearly. Is "Llama-3.1-8B-instruct" a model with instruction fine-tuning? Are these rest of ICL methods based on Llama-3.1-8B base models?
The experimental results are comprehensive. The problem studied in this paper is of interest to the deep learning community today.
This paper studies whether in-context learning can be competitive with instruction fine-tuning methods on instruction following tasks. The empirical results ablate many different components of in-context learning methods and the dataset composition. The results are comprehensive and yield several potentially useful results (e.g., how decoding parameters impact the performance). The overall conclusion is that in-context learning is worse than and not as scalable as instruction fine-tuning.
Writing is not clear. See below:
- The author should have a brief introduction to URIAL since it's heavily used in the later sections when explaining the experimental results. The current version refers to URIAL's components with the assumption that the readers know URIAL in detail. The current explanation like Line 120-126 is not enough. What do you mean by stylistic examples? What do you mean by rules? What do you mean by "begin with affirming the user queries?" I suggest adding some examples in the main paper to help the readers understand the idea.
- Similarly, what SkillMix datasets do has to be explained. 
- This paper presents many experimental findings in each section, but the messages of these findings were not highlighted. For example, Section 2.1 has the title "Systematic Evaluation of URIAL" and starts with a brief explanation of URIAL. When reading this section, I couldn't expect what you will be talking about in this section and I quickly get lost while reading a bunch of implementation details. Even after finishing the section, I couldn't get the takeaway message clearly. Section 2.2, in contrast, did a better job on explaining the results.
Yes
No ethics review needed.
- What do the authors see as the work's main contribution, apart from the empirical study of a previously introduced method?
- See other aspects mentioned above
The paper conducts a comprehensive empirical analysis of the URAIL method for aligning pre-trained LLMs via in-context learning.
It considers different datasets, LLMs and test scenarios of relevance.
The figures are well done and show interesting trends. The paper is well structured and easy to follow.
An interesting empirical study of the existing URIAL method that allows alignment via ICL. Unclear whether the contributions are significant.
I am unsure about the exact contributions of this work. It appears mainly as an extended study of the URIAL method on (a) additional models, (b) additional datasets, and (c) more hyperparameters. A relevant contribution of the work is the greedy ICL prompt search, which shows significant performance improvements (i.e., higher performance with much fewer ICL prompts, i.e., with much fewer test-time compute requirements). However, the entire analysis is again primarily based on a single dataset (MT bench). The language sometimes could be more academic, consider e.g. “suffers from some heavy overfitting”.
Yes
No ethics review needed.
For prompts ending up with varying output performances, what are their KL divergence?

If you repeat your experiments on instruct models instead of base models, what would the results be?

How does ICL cope with long-context tasks (summarization, coding, etc.) where context window limit the extent of examples?
This paper contributes valuable guidance to practitioners who often struggle between the choice of finetuning and prompting. The work is particularly useful for scenarios where obtaining training data is challenging. Also when all attentions are on instruct models, this paper shows that base models alone can be instructed and aligned, and made useful. This shall open the gate to lots of research projects too.
This paper continues on URIAL work to deep dive into the comparison of ICL vs IFT. It revealed crucial insights to improve ICL performance, i.e. high quality examples and temperate as decoding parameters. On the flip side, it also shows the shortcoming of ICL, especially on subsequent turn in conversational benchmarks.
More attribution works are needed to answer why the outcome is what it is. For example, the poor performance of the second-turn in MT benchmark definitely needs deeper dive. Is it because of the formatting or lack of examples? What caused the ICL to lose grip of the first turn? 

Considering that the authors work on small size open LLMs (except GPT-4), mechanistic study with logits or internal states shall definitely reveal more insights.
@inproceedings{
anonymous2024is,
title={Is In-Context Learning Sufficient for Instruction Following in {LLM}s?},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=STEEDDv3zI},
note={under review}
}
In-context learning (ICL) allows LLMs to learn from examples without changing their weights: this is a particularly promising capability for long-context LLMs that can potentially learn from many examples. Recently, Lin et al. (2024) proposed URIAL, a method using only three in-context examples to align base LLMs, achieving non-trivial instruction following performance. In this work, we show that, while effective, ICL alignment with URIAL still underperforms compared to instruction fine-tuning on established benchmarks such as MT-Bench and AlpacaEval 2.0 (LC), especially with more capable base LLMs. We then uncover the most relevant elements for successful in-context alignment, finding the crucial role of the decoding parameters. Based on these insights, we show that the approach of URIAL can indeed be improved by adding more, potentially carefully selected, high-quality demonstrations in context, getting closer to the performance of instruct models. Finally, we provide the first, to our knowledge, systematic comparison of ICL and instruction fine-tuning (IFT) for instruction following in the low data regime. Overall, our work advances the understanding of ICL as an alignment technique and its relationship to IFT.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Large Language Models In-context Learning Alignment
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/58b80857f3acf637abe3636e5d8d16146e5a6a70.pdf
foundation or frontier models, including LLMs
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
Is In-Context Learning Sufficient for Instruction Following in LLMs?
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission