Element-wise Attention Is All You Need

A novel element-wise attention mechanism reduces the high complexity of self-attention while maintaining comparable performance. Using squared Euclidean distance and Taylor polynomials, it achieves efficient long-sequence training with O(tLD) complexity and inference with O(tD) by leveraging an RNN structure. Unlike alternatives like linear attention or state space models, it avoids common performance degradation, making it both efficient and effective for causal and non-causal tasks.