Autoregressive Moving-average Attention Mechanism for Time Series Forecasting
Attention;Transformer;Autoregressive Moving-average;Time Series Forecasting
Yes
No ethics review needed.
1. Sensitivity of parameters not analyzed.
2. Why is each row of matrix B in the visualization basically uniform?
1. The paper introduces the ARMA model from statistics into the autoregressive attention mechanism. This structure is classic in time series analysis and can effectively handle and decouple long-term and short-term effects.
2. The ARMA attention mechanism captures short-term effects through the MA term, allowing the AR term to focus more on long-term and periodic patterns, thus balancing long-term and short-term dependencies.
3. The ARMA attention mechanism proposed in the paper introduces the MA term while maintaining the same time complexity (O(N)) and parameter scale as the underlying efficient attention model.
This paper introduces an Autoregressive (AR) Moving-Average (MA) attention structure that enhances the ability of linear attention mechanisms to capture both long-range and local temporal patterns in time series forecasting. By incorporating the ARMA structure into autoregressive attention mechanisms and using indirect MA weight generation, the proposed model improves performance on time series forecasting tasks.
1. The main advantage of a decoder-only model is that it allows for training just one model for inference at various lengths. Although there may be some cumulative error, it can still achieve state-of-the-art performance. The article's approach of "treating one-step prediction as the complete forecast" loses this advantage, as it requires training a separate model for each length.
2. The prediction length of {12, .., 96} as the main experimental setting is also unsatisfactory. The so-called fair comparison is primarily due to the fact that the input length must change according to the output length, which affects the model's applicability.
3. I don't like the large amounts of color blocks used in the experimental table.
Yes
No ethics review needed.
*Related to weaknesses*
- Could the authors perform multiple runs to obtain information on the statistical significance of the results?
- Could the author compare to the original non-deep learning ARMA models [1]?
- Could the authors perform a comparison with the two missing baselines TSMixer [2] and SAMformer [3] and mention them in the related work?
- Could the author compare the efficiency of their models also in terms of total training/inference (GPU) time?

*Potential typos*
- l269: the footnote is separated amount two pages. I think it would be better to keep it together for readability



**References**

[1] Box et al. *Some recent advances in forecasting and control*. Journal of the Royal Statistical Society, 1974

[2] Chen et al. * TSMixer: An all-MLP architecture for time series forecasting*. TMLR, 2023

[3] Ilbert et al. *SAMformer: Unlocking the Potential of Transformers in Time Series Forecasting with Sharpness-Aware Minimization and Channel-Wise Attention*. ICML, 2024
- The paper is well-written and well-structured
- The authors provided large-scale experiments with ablation studies
- Combining ARMA and attention is simple yet effective which is positive in my opinion
- The indirect MA weight generation is interesting and seems efficient with the weight-sharing
This paper proposes an autoregressive moving-average (ARMA) attention module that can be integrated into linear attention for time series forecasting.  

Contributions: The authors introduce the ARMA structure into existing autoregressive attention-based forecasting models. From a practical implementation perspective, the authors propose an indirect moving-average (ma) weight generation that maintains the complexity of the underlying attention-based models. It leads to 10 AR/ARMA models for which large-scale experiments are conducted with 12 datasets and 6 baselines. Ablation studies are also conducted.
*Performance*
- It seems that the authors did not conduct multiple runs of experiments with different random seeds. Hence, it reduces the significance of the results. While this is also the case in past works (PatchTST, Informer, Autoformer, etc.), I do believe that this is not good practice. In recent papers, e.g. in SAMformer [3],  the authors perform multiple runs which enables the statistical significance of the results and hence a better comparison.
- The authors propose a total of 10 models AR/ARMA among which only one (Lin Att + ARMA) is superior to the baselines considered and I cannot tell whether the improvement is significant.

*Computational cost and efficiency*
- I think that the autoregressive framework will not be efficient in long-term forecasting because of the H iterations needed to predict at horizon H. Hence, only comparing computational costs in terms of FLOPs does not take into account the entire process. Could the author compare the efficiency of their models also in terms of total training/inference (GPU) time?
- The authors mention in future work the extension to multivariate forecasting. I believe that the proposed models will be even less efficient in multivariate forecasting because of the autoregressive framework. Could the authors elaborate on that?

*Missing baselines*
- I am surprised the authors did not compare their approach to the original non-deep learning ARMA model [1]. Could the author compare to them?
- The authors did not compare their results to two strong recent models: TSMixer [2] and SAMformer [3]. These models are relevant given their performance and low computation cost, especially for SAMformer which is a transformer-based model. Could the author compare their approach to those models and mention them in the related work? 

*Anonymity*

The MIT Licence of the code (available at https://anonymous.4open.science/r/ARMA-attention-3437/LICENSE) contains the name of one of the authors. It should be noted that I am not familiar with it and simply mention this as a warning to advise the authors to remove the Licence in their anonymous code.
Yes
No ethics review needed.
I am concerned about the authors' claim that they mitigate error accumulation, and how the MA contribution accounts for short-term dynamics. However, this concern may be caused by confusion induced by the paper's language in section 2.5.

1) The authors state that the MA term captures short-term fluctuations. Generally, taking a moving average destroys short-term information as it filters high-frequency Fourier components. It is not even clear to me what "short-term" means: 1, 5, 10, ... ? Using a moving average will destroy information at a time-scale shorter than what I believe the authors call "short-term" but this vague language leaves the reader with relative adjectives and little quantitative grounding in time-scale.
2) The model describes $r$ as the sum of the MA term and a remaining error $\epsilon_t$. However, in the autoregressive rollout one does not have ground truth for future predictions and therefore cannot find  $\epsilon_t$ and $r$ reduces to the MA term, making $\epsilon_t=0$ for that time step. I don't understand how this mitigates the error accumulation.
This work shows strong results against other models and applies an autoregressive decoder-only method, which I believe has more potential in time-series forecasting. The authors do a good job explaining other types of attention and make an attempt to cast their work relative to other types of attention. This context is both very helpful and gives the reader confidence, but the paper lacks some clarity to fully realize these comparisons (described later). In my opinion, the underlying work, results, and usage of the decoder-only architecture can be sufficient for publication here but not in its current state.
The authors employ an autoregressive decoder only transformer architecture to time-series data. The autoregressive nature is supplemented with a moving average "corrector" component inspired by statistical temporal analysis methods. The authors show that their method outperforms other methods on short-range time predictions.
The paper suffers from a lack of clarity and detail. 

1) Section 2.5 needs to be completely rewritten with more detail and explanation. This section is crucial to understanding the rest of the paper and deserves more space and detail. It is unclear to me how the MA term fits into all of this. While this section attempts to explain it, it is unclear. For example, "In an ARMA model, the MA term captures short-term fluctuations." This is the fits time MA is mentioned and the authors already say that MA is in ARMA, implying the reader should know the architecture a prior. 
2) There is a lot of time-series work, and much of it uses longer time horizon predictions (as the authors mentions). However, it was not clear to me that this work focuses on shorter time horizons until far later in the paper. The authors need to make this more clear.
3) The authors need to motivate why predicting on shorter time-scales is more important since other time-series methods can predict further into the future.
4) Figure 3 looks important but is not mentioned or used in section 2.5. This seems like a misuse of an important figure. 
5) Please align the vectors in Eq. 3
6) It is very hard to read the labels in Fig. 4
7) In general it is difficult to read some of the tables, the text is fairly small.
Yes
No ethics review needed.
See my comment above
The idea is clever, and experimentally, all the method's aspects seem well analyzed: from the added computational load perspective to the combination of the MA term with different attention mechanisms. The supplementary material offers a complete experimental evaluation.
The author introduces a moving average term into the autoregressive attention model for linear attention mechanisms, indirectly computed using a linear RNN to collect all keys and values at once.
Some issues from which I would like clarification/further insights:

- You stress the importance of the MA in capturing short-term effects and letting the attention model include long-term effects. Wouldn't it be better to exploit the signal's spectral properties to address that? Two recent approaches that consider this idea (among many others in previous years) are Fredformer (https://arxiv.org/abs/2406.09009) and TSLANet: (https://arxiv.org/pdf/2404.08472). I think these two papers should be included in the comparison.

- You state that error accumulation is avoided in section 2.2. by jointly predicting signal periods of length $L_P$ (instead purely next token prediction of the signal). Is that a reasonable assumption?  When you need to forecast a signal with length above $L_P$, you have to feed back your estimate to the first $L_P$ terms back to the decoder to estimate the next signal values...

- Why do you only show your results using MSE? A quantile loss is definitely more descriptive and reliable, and it should be included. 

- As you say, you have not explored combining the channel-independent ARMA Transformer with multivariate forecasting models to improve forecasting. Namely, you fit one model per channel. Have you done the same for the rest of the baselines? I would say this is not fair if the baselines can easily integrate the channels into the same model and exploit interdependencies.
@inproceedings{
anonymous2024autoregressive,
title={Autoregressive Moving-average Attention Mechanism for Time Series Forecasting},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=Z9N3J7j50k},
note={under review}
}
We propose an Autoregressive (AR) Moving-average (MA) attention structure that can adapt to various linear attention mechanisms, enhancing their ability to capture long-range and local temporal patterns in time series. In this paper, we first demonstrate that, for the time series forecasting (TSF) task, the previously overlooked decoder-only autoregressive Transformer model can achieve results comparable to the best baselines when appropriate tokenization and training methods are applied. Moreover, inspired by the ARMA model from statistics and recent advances in linear attention, we introduce the full ARMA structure into existing autoregressive attention mechanisms. By using an indirect MA weight generation method, we incorporate the MA term while maintaining the time complexity and parameter size of the underlying efficient attention models. We further explore how indirect parameter generation can produce implicit MA weights that align with the modeling requirements for local temporal impacts. Experimental results show that incorporating the ARMA structure consistently improves the performance of various AR attentions on TSF tasks, achieving state-of-the-art results. The code implementation is available at the following link: https://anonymous.4open.science/r/ARMA-attention-3437.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Attention Transformer Autoregressive Moving-average Time Series Forecasting
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/3c6bc76f1a2a3d179b49eec678ac69611a72e660.pdf
learning on time series and dynamical systems
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
Autoregressive Moving-average Attention Mechanism for Time Series Forecasting
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission