Large Language Models Can Be More Robust Multiple Choice Selectors Through Attention Intervention
Large Language Models; Reasoning; In-context learning
Yes
No ethics review needed.
1. Why does context option bias for MCQ a practical problem?

2.  How does the performance scale with more examples?

3. What happens if you use a custom prompt message?

4. What is the impact of model size on the problem?

5. How does the performance change w.r.t. different $K_l$ and $K_h$? How did you choose these parameters?

6. Can you improve the baseline by finding the best temperature?

7. Why does the performance of Llama-3 and Gemma different from official numbers?
1. The custom temperature per attention head is a simple but affective approach for detecting context option bias.

2. Experimental results indicate success with better detection and improved performance.

3. Transferability to other tasks indicate that these heads are feature of the model and pre-training rather than downstream domain. This is an interesting finding.
This paper proposes a method to interpret which attention cells are responsible for “context option bias” — a form of bias that occurs when an LLM inclines to generating one option more than others in an in-context learning scenario. The authors propose a difference metric that measures the change in attention values when biased or ordinary examples are used with in-context learning. This metric is also used to detect top-layers that are the most important for context option bias. They also propose a solution to alleviate the impact of bias: scaling attention values of the detected cells using a custom temperature. Experimental results suggest that the proposed method improves MMLU performance and detected attention heads transfer to other domains such as CEVAL. The detected heads are robust to different number of choice or demonstrations but sensitive to the particular biased option.
There are a few concerns that I detailed below.

1. I think a better motivation on why context option bias for MCQ is a practical problem is needed. Given that options in in-context examples can be shuffled freely, this needs a better motivation.

2. What is the minimal number of examples for successful detection of bias? More specifically, how does the performance scale with more examples?

3. What if you add a custom prompt message that explains that “there are multiple options, the answer can be any one of them, and no particular option such as A or B is more likely.” Would this be an affective strategy to steer the attention heads?

4. How does the problem change with increasing model size? Are larger models more or less susceptible?

5. Equation 2 should be $y=argmax$.

6. How did you choose $K_l$ and $K_h$? How does the performance change as $K_l$ and $K_h$ increases?

7. Previous work has used different temperatures for different domains (see [1] for one such work) that are not necessarily 1.0. Can you run a search over different temperatures for the baseline and report the best performing model?

8. Can you explain why the performance of Llama-3 and Gemma models are very different from official numbers from respective papers?

9. Can you explain what do you mean by confidence in Section 4.5?

10. Caption of the Figure-5 is not grammatically sound. Please rewrite.

[1] Large Language Monkeys: Scaling Inference Compute with Repeated Sampling. Bradley Brown and Jordan Juravsky and Ryan Ehrlich and Ronald Clark and Quoc V. Le and Christopher Ré and Azalia Mirhoseini.
Yes
No ethics review needed.
Did you experiment with other recent llms, such as Llama 3.2, Mistral, Qwen 2, do they suffer from similar biases? Is the proposed method effective in these cases? 

Why reducing the bias in such artificially biased scenario will lead to better performance in real-world applications?

Did you try other options of reducing the bias, like targeted fine-tuning?

Why the standard deviations in the experiments were not reported?

How to tune the hyperparameters of the method such as T? the performance seems to depend heavily on its value (see table 1)
-Paper addresses the weaknesses in large language models related with content bias in multiple-choice questions task (MCQ).

-The paper claims that the proposed method is demonstrated empirically to reduce the content bias in MCQ task on MMLU, although the standard deviations in the experiments are not  reported.
The paper proposes  a method  aiming at mitigating context option bias in multiple-choice questions taks, by comparing the LLMs’ outputs for biased and ordinary prompts in order to localize attention heads associated with the context option bias. Attention scaling interventions is then employed. Experiments on several dataset are described in order to validate the method, although the standard deviations are not reported.
1)The paper provides no theoretical explanation for why only specific attention heads contribute to bias or why scaling the specific attention heads mitigates the bias effectively. This lack of theoretical grounding may affect the reproducibility and understanding of the method, making it difficult to predict its behavior across other models.

2)The standard deviations are not reported in the experiments. Especially in the view of the small differences in several important experiments, this makes impossible to evaluate the method empirically.

2)Narrow focus on attention scaling as the intervention technique. The paper uses exclusively the attention scaling to address bias without exploring other possible intervention methods, such as layer-based adjustments or targeted fine-tuning. This narrow approach may miss alternative solutions that could yield more robust or efficient debiasing effects.

3)Recent large language models, such as  for example Llama 3.2, Mistral, Qwen 2 were not used in experiments, do they suffer from similar biases? Is the proposed method effective in these cases?

4)Reliance on artificially biased prompts. The study constructs heavily biased demonstrations to amplify context option bias and identify attention heads associated with this bias. This contrived setup may not accurately reflect the subtler biases encountered in real-world applications, potentially limiting the generalizability of the findings. It is not clear why reducing the bias in the artificially biased scenario will lead to increased performance in real-world applications. 

5)Sensitivity to hyperparameter choice. The method has hyperparameters, such as the scaling factor T for attention adjustments, but the paper  analysis of how sensitive the model's performance is to these choices is limited. Without detailed guidance for these hyperparameters, reproducing the results and applying the method to new datasets or models would require extensive tuning, which can be  resource-intensive.

6)Limited evaluation of cross-domain generalizability.  The evaluation of cross-dataset robustness is limited to certain datasets and tasks, where the reported small change is impossible to validate in the absence of  standard deviations reported. A more comprehensive assessment across a broader range of domains would be necessary to confirm the method’s generalizability, especially given the diverse biases in data sources.

7)The readability and clarity of the paper can be improved. The paper often employs long sentences which can obscure the main points. Sentences like 'To better observe the LLMs’ behavior when affected by the context option bias, we deliberately use demonstrations with obvious context option bias for MCQ to amplify the effect' are long and somewhat repetitive. Table 2 is discussed briefly without highlighting the most important findings or explaining how they support the claims. Redundant phrases and wordiness occasionally affect readability. For example, phrases like 'To better excavate the reasons behind this problem and resolve it...' could be made more concise. 'One of effective methods' requires 'the' article here; '...significantly degrade LLMs’ performance' as  'context option bias' is singular, so 'degrade' should be 'degrades.'  Vague pronoun references, in a few instances, pronouns (like 'it' or 'this') refer ambiguously to previous sentences, making the meaning less clear. 'The experiments find that... This inspires us to test...' — here, "This" could refer to several prior ideas, so specifying 'This result' or 'This observation' would improve clarity.

8)The captions under Figure 8 are somewhat misleading, how to read the frequency of the localized field from picture (c)?
Yes
No ethics review needed.
See weaknesses
_ Proposed approach is novel and the problem of biased in-context examples in important
- Paper is clearly written and easy to understand
- Experiments are thorough, including different models and tasks
- Results show that the approach leads to improved performance and is robust to the style of the input and the dataset
Paper proposes a new approach to identify and intervene with attention heads in order to minimize biases from in-context examples
- Results mostly show 1-3% improvement, which is nice, but not a huge improvement
- Experiments are focused on multiple choice. It would be great to show that the proposed method is able to generalize to more open ended question answering tasks, which are more representative of realistic tasks that LLMs are used for
- The proposed method requires both original and biased versions of in-context examples. However, in more open ended question, LLMs may still suffer from biased in-context examples, but it may not be clear what the bias is. Is there a way to not required both biased and unbiased versions of in-context examples in the method?
Yes
No ethics review needed.
1. Identifying the attention heads most associated with context option bias requires multiple rounds of sampling, which could add computational complexity in real-world applications (and higher than the PriDe baseline). While this overhead is justified within the paper’s scope, discussing potential optimizations or alternative approaches might help broaden CoLo’s applicability, especially for larger-scale implementations.
2. While the concept of CoLo is sound, the process for selecting and intervening on attention heads may seem somewhat convoluted to readers not already familiar with the technical nuances of attention heads and scaling interventions. Additional clarifications or illustrative examples could enhance the accessibility of the proposed approach.
* I appreciate the focus on context option bias, a previously-overlooked form of bias that can impact LLM performance in MCQ tasks. The design of CoLo is also well-motivated.
* The paper provides extensive empirical evidence that shows improvements in MCQ accuracy and recall standard deviation. The authors also consider both zero-shot and few-shot scenarios and the cross-dataset setting, enhancing the generalizability of their findings.
This paper presents an approach, termed CoLo, to address the bias in multiple-choice question (MCQ) selection that arises from context option bias in large language models (LLMs). CoLo mitigates context option bias by intervening on specific attention heads that are highly sensitive to biased in-context demonstrations. By selectively scaling attention in these heads, CoLo reduces the model's predisposition to biased options, showing improvements in MCQ accuracy across several datasets. The authors also conduct extensive experiments to validate the cross-domain robustness of their method, indicating that CoLo's effect generalizes well across various MCQ scenarios.
I would like to see evaluation with more larger models, as the current experiments are conducted with at most 7B/8B-level models. So I am not fully convinced about CoLo's scalability. For example, how CoLo would perform with 30B/70B-level, or MoE models (these experiments may be conducted with Qwen2/Qwen2.5 models, as LLaMA-3 does not have too many variants)? Also, what the specific GPU hardware and hours will be needed for various-sized models? The lack of experiments and analysis on larger models may limit the broader impact and implications of the proposed method.
A method for localizing and mitigating option bias in large language models.
@inproceedings{
anonymous2024large,
title={Large Language Models Can Be More Robust Multiple Choice Selectors Through Attention Intervention},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=kauK28525D},
note={under review}
}
Multiple-choice question (MCQ) is a common task for evaluating large language models (LLMs). LLMs' performance on MCQ is often affected by various biases. Previous research has extensively examined the impact of inherent option bias on MCQ predictions, where this bias refers to a preference for a specific option ID token introduced during the model's training. However, in an in-context learning scenario, few-shot prompting can also introduce a form of bias, known as context option bias. This occurs, for instance, in extreme cases where all demonstration answers are consistently option A, in which case LLMs may predict A for the given question whatever the question is. Context option bias can significantly degrade LLMs' performance. To better observe the LLMs' behavior when affected by the context option bias, we deliberately use demonstrations with obvious context option bias for MCQ to amplify the effect. The results indicate that certain attention heads in LLMs are particularly sensitive to context option bias. Motivated by this observation, we propose our approach, CoLo, to address this issue. First, using samples with ordinary and biased demonstrations as input, CoLo compares the outputs of two types of inputs and localizes attention heads sensitive to context option bias through sequential interventions. Then, we propose an attention scaling-based method to intervene in the identified attention heads during the inference stage, thereby mitigating the impact of context option bias on the LLMs’ predictions. Experimental results demonstrate that CoLo effectively alleviates the impact of context option bias and improves the LLM's robustness on MCQ tasks.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Large Language Models; Reasoning; In-context learning
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/8b3300eadbacf1afe6148f257eb3227483f4353e.pdf
interpretability and explainable AI
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
/attachment/03c797a19650d55ea218b5b742509fd02401c0e8.zip
Large Language Models Can Be More Robust Multiple Choice Selectors Through Attention Intervention
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission