From Few to Many: Enhancing Many-Shot In-Context Learning with Optimized Example Selection and Expansion
many-shot;in-context learning;large language models
Yes
No ethics review needed.
Q1 In reference to Figure 1, does bottom_k refer to selecting examples that are less similar to the test case? Why does bottom_k achieve performance comparable to top_k as the number of demonstrations increases? Additionally, why does bottom_k outperform the all-examples approach?
- The proposed framework integrates example selection with LLM-generated examples to better align with real-world applications. 

- The paper validates this approach through extensive experiments conducted across a wide range of tasks.
This paper addresses the many-shot in-context learning (ICL) paradigm. In contrast to few-shot learning, many-shot ICL leverages a larger number of demonstration examples to better exploit long-context capabilities. Initially, the paper demonstrates that selecting influential demonstration examples remains valuable, even in long-context settings. Building on this observation, the authors propose an algorithm that alternates between two key steps: an optimization step, which employs Bayesian optimization to identify influential example sets, and a generation step, which reuses these sets to automatically extend reasoning paths, thereby facilitating the transition back to the many-shot regime.
The validation of the findings appears overly simplistic. For instance, as model capabilities scale, the order and number of demonstration examples may have diminishing importance in affecting in-context learning (ICL) performance. The authors should reconsider their conclusions by re-evaluating their results on less advanced LLMs, providing a more comprehensive understanding of the interplay between long-context utilization and demonstration selection. Without this analysis, the conclusion may be over-claimed to specific models.

Lack of baselines. In the experiments, the authors did not compare their approach with existing learning-based strategies and learn-free ones for demonstration selection. 

The writing of the paper requires further refinement. Sections 3 and 4 are challenging to read and follow, despite the method itself being relatively straightforward. For instance, the description of the different variants could be more appropriately placed in the Experimental Setting section for better clarity.
Yes
No ethics review needed.
I don't fully understand the claim that BRIDGE is tailored to many-shot ICL. I feel it also applies to few-shot ICL?
- Many-shot ICL is an important problem from both the capability and safety angles. 
- This paper shows that even for many-shot ICL, good examples are still necessary. 
- The proposed method is a natural extension of the STaR framework to ICL. Given the hardness of optimizing ICL example selection, the proposed optimizer looks like a good contribution.
- Experimental results on BBH look strong, and small but non-trivial improvement are shown on math (MATH and GSM8K) and text-to-sql (BIRD) datasets.
This paper studies the problem of how to improve many-shot in-context learning (ICL). The first observation is that the quality of ICL examples can be more important than the quantity -- a carefully selected subset of in-context learning examples can outperform the whole set. 

Based on this observation, this paper naturally moves to the question of how to improve the quality of ICL examples. The key idea is to iteratively select helpful examples from both the seed examples and those generated by the model itself, similar in spirit to STaR but applied to ICL instead. Specifically, in each iteration, the model is prompted to generate both reasoning steps and the answer. Only those with correct answers are kept. Then a Bayesian Optimizer is used to select the best ICL examples in this iteration, which are used as ICL examples for generation in the next iteration.

Results on BBH show that the proposed method largely outperforms baseline methods. Small improvements are achieved on MATH and GSM8K. A large improvement is observed on the BIRD-Challenging subset.
- The experiments are done only with Gemini models on text tasks. It would be interesting to see the performance on GPT-4o / o1 models and multimodal tasks. 
- Variance is not reported for MATH, GSM8K, and BIRD. Given the relatively small improvement on these datasets, it's unclear if the gain is significant.
- As the method optimize the performance on a dataset/task, it would be interesting to see the performance under distributional shifts. For example, does the gain on GSM8K transfer to GSM-symbolic?
Yes
No ethics review needed.
Can you please provide the baseline when a model conditions on all available examples but with all examples using model generated reasoning? This baseline can decouple the importance of subselecting a subset of examples and influence of having model-generated reasoning, and strengthen your claims if, indeed, you need both rather than only model-generated reasoning. The recent works [1, 2] can be used to ensure that a model is constrained to generate reasoning given the input and the ground truth answer.

[1] Hu et al. Amortizing intractable inference in large language models. ICLR 2024

[2] Zhao et al. Probabilistic Inference in Language Models via Twisted Sequential Monte Carlo. ICML 2024
* Paper studies many-shot in-context learning, the important novel prompting technique that is used to significantly improve LLMs' predictions. Given that, this technique was made available only recently with the rise of long-context LLMs, studying its behaviour is important topic that can be relevant to broad community.

* The proposed approach is simple and algorithmically applicable to both close-source and open-source models.
The paper studies the many-shot in-context learning using Gemini family of models on BigBenchHard (BBH). Specifically, the authors demonstrate that identifying a subset of influential examples reinforced with model-generated reasoning can significantly outperform naively using all available examples. They also propose BRIDGE, a simple approach that instantiates this finding to iteratively seeking for the subset of examples and refining model-generated reasoning for these examples. BRIDGE is evaluated on the subset of BBH tasks and the substantial gains over the considered baselines are demonstrated.
* The important baseline might be missing to ensure the strength of the claims, please see Questions section.

* I would also ask for including similar analysis for other LLMs beside close-source ones like Gemini. At least on the subset of the considered tasks, to see whether the conclusions generalize to other models.
Yes
No ethics review needed.
Please refer to the "weakness".
1. Overall, this paper is well writtern and clear to follow.
2. The proposed method is well-motivated through experiment results (Figure 1 and Figure 2). The observation that many-shot ICL could be driven by few high-performing examples is interesting.
3. The proposed method shows decent performance compared to the baseline.
4. The method is evaluated on various  challenging datasets
This paper presents a study on the performance of many-shot in-context learning on NLP tasks that require reasoning process (the reinforced ICL setup). The authors first define a importantce scoring function of input examples based on estimated input gradient, and observe that 1) many-shot ICL could be driven by a few high-performing examples, 2) scaling demonstration examples could still be beneficial with improved quality of reasoning. Motivated by these observation, the authors propose a method called BRIDGE to improve many-shot ICL in an iterative manner. Experiments of Gemini across a diverse set of tasks show the effectiveness of BRIDGE.
1. The analysis is limited to reinforced ICL setting that only considers reasoning-heavy tasks. It's not clear what many-shot ICL would perform on simpler tasks that do not heavily relies on reasoning. Also, could BRIDGE anyway be helpful on simpler tasks?
2. The experiments are only evaluated with Gemini. What would be the performance of proposed method on open-source methods like Llama-3.1? More importantly, for open-source model, would there exists a better way of importance attribution? Also, for base model where the reasoning ability is limited, would the proposed method also be helpful?
3. Lack of the analysis on the computation cost. What's the computational cost of optimize and generate step of BRIDGE, respectively? 
4. The definition of importance score function is well-justified. In section 2 Line 153, the importance of each example is estimated by the averate performance of randomly sampled demonstrations on the validation set. The variance would be huge considering different choices of subsets and orders. Have the author performed analysis on the varaince of importance score?
5. Importance attribution is not well justified. In section 2 Line 153, it's not clear how importance score is actually estimated and why it would make sense.
6. The description of "optimzie" step is not clear (Line 261). The text in page 6 is generally unclear to me. For example, what does "TCM" stand for in algorithm 2 step 6? How does the problem become a "bi-objective" problem and how the optimize step is connected to "Bayes optmization"?

Minor issue: ")" is missing in Line 149
@inproceedings{
anonymous2024from,
title={From Few to Many: Enhancing Many-Shot In-Context Learning with Optimized Example Selection and Expansion},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=JBXO05r4AV},
note={under review}
}
Recent advances in long-context large language models (LLMs) have led to the emerging paradigm of many-shot in-context learning (ICL), where it is observed that scaling many more demonstrating examples beyond the conventional few-shot setup in the context can lead to performance benefits. However, despite its promise, it is unclear what aspects dominate the benefits and whether simply scaling to more examples is the most effective way of improving many-shot ICL. In this work, we first provide an analysis on the factors driving many-shot ICL, and we find that 1) many-shot performance can still be attributed to often a few disproportionately influential examples and 2) identifying such influential examples ("optimize") and using them as demonstrations to regenerate new examples ("generate") can lead to further improvements. Inspired by the findings, we propose BRIDGE, an algorithm that alternates between the \textit{optimize} step with Bayesian optimization to discover the influential sets of examples and the \textit{generate} step to reuse this set to expand the reasoning paths of the examples back to the many-shot regime automatically. On two state-of-the-art long-context Gemini models of different sizes, we show \ours led to significant improvements across a diverse set of tasks including symbolic reasoning, numerical reasoning and code generation.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
many-shot in-context learning large language models
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/a72019adee76577911b984963fb2c19b73187884.pdf
foundation or frontier models, including LLMs
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
From Few to Many: Enhancing Many-Shot In-Context Learning with Optimized Example Selection and Expansion
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission