PEAR: Position-Embedding-Agnostic Attention Re-weighting Enhances Retrieval-Augmented Generation with Zero Inference Overhead
Retrieval-Augmented Generation;Large Language Model;Context Awareness;Re-weighting Attention Heads
We have found some typos.
I have read and agree with the venue's withdrawal policy on behalf of myself and my co-authors.
@misc{
tan2024pear,
title={{PEAR}: Position-Embedding-Agnostic Attention Re-weighting Enhances Retrieval-Augmented Generation with Zero Inference Overhead},
author={Tao Tan and Yining Qian and Ang Lv and Hongzhan Lin and Songhao Wu and yongbo wang and Feng Wang and Jingtong Wu and xin lu and Rui Yan},
year={2024},
url={https://openreview.net/forum?id=XpMDSArfGw}
}
Large language models (LLMs) enhanced with retrieval-augmented generation (RAG) have introduced a new paradigm for web search. However, the limited context awareness of LLMs degrades their performance on RAG tasks. Existing methods to enhance context awareness are often inefficient, incurring time or memory overhead during inference, and many are tailored to specific position embeddings. In this paper, we propose \textbf{P}osition-\textbf{E}mbedding-\textbf{A}gnostic attention \textbf{R}e-weighting (\textit{PEAR}), which enhances the context awareness of LLMs with zero inference overhead. Specifically, on a proxy task focused on context copying, we first detect heads which suppress the models' context awareness, thereby diminishing RAG performance. To weaken the impact of these heads, we re-weight their outputs with learnable coefficients. The LLM (with frozen parameters) is optimized by adjusting these coefficients to minimize loss on the proxy task. During inference, the optimized coefficients are fixed to re-weight these heads, regardless of the specific task at hand. Our proposed \textit{PEAR} offers two major advantages over previous approaches: (1) It introduces zero additional inference overhead in terms of memory usage or inference time, while outperforming competitive baselines in accuracy and efficiency across various RAG tasks. (2) It is independent of position embedding algorithms, ensuring broader applicability.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
~Tao_Tan6 ~Yining_Qian1 ~Ang_Lv1 ~Hongzhan_Lin2 ~Songhao_Wu1 ~yongbo_wang4 ~Feng_Wang8 ~Jingtong_Wu1 ~xin_lu17 ~Rui_Yan2
Tao Tan Yining Qian Ang Lv Hongzhan Lin Songhao Wu yongbo wang Feng Wang Jingtong Wu xin lu Rui Yan
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Retrieval-Augmented Generation Large Language Model Context Awareness Re-weighting Attention Heads
I certify that there is no acknowledgement section in this submission for double blind review.
tan|pear_positionembeddingagnostic_attention_reweighting_enhances_retrievalaugmented_generation_with_zero_inference_overhead
/pdf/cf2802bf03425176286a95caa7b83c966e696196.pdf
foundation or frontier models, including LLMs
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
No
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
/attachment/95848afc4dd740a4310e96b800d979d1dbddeb02.zip
PEAR: Position-Embedding-Agnostic Attention Re-weighting Enhances Retrieval-Augmented Generation with Zero Inference Overhead
ICLR 2025 Conference Withdrawn Submission
ICLR.cc/2025/Conference/Withdrawn_Submission