Generalized Attention Flow: Feature Attribution for Transformer Models via Maximum Flow
Attention Flow;Feature Attributions;Transformers;Barrier Regularization;Maximum Flow
Margin violation
Submission Desk Rejected by Program Chairs
This paper presents Generalized Attention Flow, an extension of Attention Flow that uses attention weights, their gradients, maximum flow, and the barrier method to define information tensors for generating feature attributions in Transformer models.
@misc{
aghmiyouni2024generalized,
title={Generalized Attention Flow: Feature Attribution for Transformer Models via Maximum Flow},
author={Behrooz Azarkhalili Aghmiyouni and Maxwell Libbrecht},
year={2024},
url={https://openreview.net/forum?id=2RcTuBc4mA}
}
This paper introduces Generalized Attention Flow, a novel feature attribution method for Transformer models that addresses the limitations of existing approaches. By generalizing Attention Flow and substituting attention weights with an arbitrary Information Tensor, the method leverages attention weights, their gradients, maximum flow, and the barrier method to generate more accurate feature attributions. The proposed approach demonstrates superior theoretical properties and resolves issues associated with previous methods that rely solely on simple aggregation of attention weights. Comprehensive benchmarking in NLP sequence classification tasks reveals that a specific variant of Generalized Attention Flow consistently outperforms state-of-the-art feature attribution methods across most evaluation scenarios, offering a more accurate explanation of Transformer model outputs.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
~Behrooz_Azarkhalili_Aghmiyouni1 ~Maxwell_Libbrecht1
Behrooz Azarkhalili Aghmiyouni Maxwell Libbrecht
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Attention Flow Feature Attributions Transformers Barrier Regularization Maximum Flow
I certify that there is no acknowledgement section in this submission for double blind review.
aghmiyouni|generalized_attention_flow_feature_attribution_for_transformer_models_via_maximum_flow
/pdf/77bf7201d7f76cb90754860c7a126c14ffb9c5ba.pdf
interpretability and explainable AI
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
/attachment/f3d613932cdd85ae594625205c0d411887360247.zip
Generalized Attention Flow: Feature Attribution for Transformer Models via Maximum Flow
ICLR 2025 Conference Desk Rejected Submission
ICLR.cc/2025/Conference/Desk_Rejected_Submission