UniTST: Effectively Modeling Inter-Series and Intra-Series Dependencies for Multivariate Time Series Forecasting
Multivariate Time Series Forecasting
Yes
No ethics review needed.
- Could the authors further expand the search range for hidden dimensions to evaluate all models' performance fairly? Also, Please publicly disclose the optimal parameters for all models across various tasks as determined through validation sets (batch size, hidden dimensions, etc.). This is crucial to validate the performance of UniTST and mitigate any impact from selective parameter choices.

- I noticed the experimental results in Table 6, which seem to show minimal improvement compared to iTransformer in many cases. Could the authors, after addressing question 1, provide new experimental results, analyze the reasons behind them, and conduct a comparative analysis of the efficiency (time/memory) of these two methods?

-  Can the authors provide a case study of a variable prediction curve?
**S1:** The multivariate time series forecasting problem focused on in this paper is worthy of investigation.

**S2:** The author's motivation for the study makes sense.

**S4:** The paper is well-written and well-organized, and the content reads smoothly.
This paper addresses the issue that previous attention-based methods were unable to model both temporal and channels simultaneously. It proposes a flattening approach and draws inspiration from the attention mechanisms of ETC and Crossformer to reduce the complexity of attention calculations. Finally, it attempts to demonstrate the effectiveness of the method through experiments.
**W1:** The authors' model design lacks innovation. While flattening patches makes sense, there is a lack of innovation in the model design, as the Transformer architecture used does not appear to offer any significant novelty. The approach to Attention with dispatchers and the setup resembling ETC[1] and Crossformer[2] are nearly identical.

[1] Ravula, A., Alberti, C., Ainslie, J., Yang, L., Pham, P. M., Wang, Q., ... & Fisher, Z. (2020, June). ETC: Encoding long and structured inputs in transformers. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP).

[2] Zhang, Y., & Yan, J. (2023, May). Crossformer: Transformer utilizing cross-dimension dependency for multivariate time series forecasting. In The eleventh international conference on learning representations.

**W2:** The authors' lack of thorough research on past methods is concerning. While authors took note of iTransformer from ICLR 2024, they failed to consider contemporary state-of-the-art methods such as TimeMixer[3], FITS[4], and ModernTCN[5]. Furthermore, this still lacks a comparison with GNN-based methods like CrossGNN[6] and FourierGNN[7] from NeurIPS 2023.

[3] Wang, S., Wu, H., Shi, X., Hu, T., Luo, H., Ma, L., ... & ZHOU, J. (2024). TimeMixer: Decomposable Multiscale Mixing for Time Series Forecasting. In The Twelfth International Conference on Learning Representations.

[4] Xu, Z., Zeng, A., & Xu, Q. (2024). FITS: Modeling Time Series with $10 k $ Parameters. In The Twelfth International Conference on Learning Representations.

[5] Luo, D., & Wang, X. (2024). Moderntcn: A modern pure convolution structure for general time series analysis. In The Twelfth International Conference on Learning Representations.

[6] Huang, Q., Shen, L., Zhang, R., Ding, S., Wang, B., Zhou, Z., & Wang, Y. CrossGNN: Confronting Noisy Multivariate Time Series Via Cross Interaction Refinement. In Thirty-seventh Conference on Neural Information Processing Systems.

[7] Yi, K., Zhang, Q., Fan, W., He, H., Hu, L., Wang, P., ... & Niu, Z. FourierGNN: Rethinking Multivariate Time Series Forecasting from a Pure Graph Perspective. In Thirty-seventh Conference on Neural Information Processing Systems.

**W3:** The experimental comparisons are not enough insufficient. The methods mentioned in W2 were also not compared by the authors, therefore, it cannot be concluded that UniTST achieves SOTA performance.

**W4:** The lack of details in reproducibility.
Yes
No ethics review needed.
Training was conducted using an A100 40GB GPU, which, under typical conditions, rarely runs out of memory—except when working with large models such as TimesNet on the Traffic dataset. However, the ablation study does not specify the batch size used, also there are questions about the choice of memory usage as a comparative metric. A detailed explanation is needed to clarify why memory usage was selected as a benchmark for this experiment and why one out of four ablation experiments resulted in an out-of-memory (OOM) error.

The motivation for this study is not clearly defined and appears somewhat unconvincing. Although the results from the experiment in Figure 3 are cited as the primary source of motivation, the rationale behind conducting the experiment in Figure 3 itself is unclear. Most existing models that aim to capture inter-variable and intra-variable dependencies in multivariate data do not employ patch operations. Thus, the introduction of patching in the experiment warrants further explanation. A more rigorous clarification of the motivation, particularly regarding the choice to incorporate patch operations, would strengthen the foundation and relevance of this study.
The paper clearly outlines the limitations of existing Transformer models in capturing both inter-variable and intra-variable dependencies within multivariate time series data. To address this issue, it introduces the UniTST model.

The proposed UniTST model employs a unified attention mechanism alongside a scheduler module to simultaneously capture inter-variable and intra-variable dependencies. This innovative design effectively enhances the handling of multivariate time series data.

The experimental results presented in the paper are reproducible, reliable, and credible.
UniTST is designed to address existing models' limitations in capturing complex dependencies across variate and temporal dimensions in multivariate time series (MTS) data. The authors argue that previous Transformer models have fallen short in simultaneously capturing inter-variate (between different series) and intra-variate (within the same series) dependencies, critical for accurate forecasting in real-world data.

The authors introduce a unified attention mechanism within UniTST that operates on flattened patch tokens, allowing the model to directly and explicitly learn the intricate inter-series and intra-series dependencies. To manage the increased complexity associated with a large number of variates, a dispatcher module is integrated into the model, reducing the computational complexity from quadratic to linear, thus making the model scalable.

The paper contributes to the field by highlighting the importance of inter- and intra-variate dependencies in MTS forecasting, proposing a simple yet effective model architecture to capture these dependencies, and empirically demonstrating its superiority over existing methods. The findings emphasize the necessity of simultaneously modelling variate and temporal dynamics in multivariate time series analysis.
The motivation behind the study is not clearly articulated, which makes it difficult to fully understand the underlying rationale and significance of the research problem. Additionally, the proposed approach lacks sufficient novelty, as it does not introduce substantially new concepts or techniques compared to existing methods. Strengthening the motivation and highlighting the unique contributions of the work would enhance its originality and impact within the field.

In the ablation study, the effectiveness of the dispatcher is evaluated based on memory usage, an approach that is relatively uncommon. While comparing the memory consumption of the dispatcher module with other modules within the model provides useful insights,
However, it is not sufficient to conduct ablation experiments on only one component within the model. It is also essential to assess the whole model's memory impact relative to other models. A more rigorous evaluation would involve additional comparisons with state-of-the-art (SOTA) models, focusing on computational cost and model parameter size both before and after integrating the dispatcher. This broader comparison would offer a clearer understanding of the dispatcher’s role and efficiency within the model.
Yes
No ethics review needed.
1. What is the value of t' corresponding to the correlations in Figure 3?
2. Since the router operation, that is, the Dispatcher proposed in the article, is used, it may not be possible to directly visualize the attention weights between variables learned by the model to correspond to the correlation calculated by Definition 1. Then, can you provide the correlation value calculated for the model's prediction results and compare it with the correlation based on real data given in Figure 3? Does the model truly capture the dependencies across different variates and different times that you proposed or other fitting results?
3. It is recommended to provide code for reproducibility.
Experiments have been conducted on the major datasets currently available. The experiments are relatively sufficient and the analysis is comprehensive, including hyperparameter analysis and efficiency analysis.
It is recommended to add some visualized prediction results and compare them with the state-of-the-art. This is a relatively common way of presenting results.
The article proposes a dependencies across different variates and different times and presents a corresponding Transformer modeling method. However, this greatly increases the number of tokens. To reduce complexity, a Dispatch method is proposed.
1. For the current research situation, this contribution is relatively ordinary. There are many almost identical practices in the past. For example, Different sEnsors at Different Timestamps (DEDT) in https://arxiv.org/abs/2309.05305  is completely the same as "across different variates and different time" in the second line of the third paragraph of the Introduction. Cross-correlation coefficient in https://arxiv.org/abs/2401.17548 is almost same with the Definition 1,  just with different names. I think the explanation of the problem and the drawings are not as clear as those in past articles. 
At the end of the second paragraph of the Introduction, the problems in the two stages are mutually influential. I think there is a lack of experimental proof for the problem. Moreover, the two-stage method in the past https://arxiv.org/pdf/2402.19072 is not inferior to UNITST in terms of effect.
3. The Dispatcher is also the same as the router in Crossformer . There is no innovation.
4. No code is provided for reproducibility testing, the authenticity of the experimental results is reserved.
5. Incidentally, the article uses the template of ICLR 2024. When reviewing, it is not convenient to locate the exact line.
Yes
No ethics review needed.
The author flattens all patches from different variates into a unified sequence. I wonder whether the sequence is (1) normalized, (2) univariate or multivariate. Furthermore, if the input dimension is high, the unified sequence may be too long. 

What is the difference of constructing a unified sequence and simple concatenation? It seems very similar to me. 

How does the author determine the order of raw input series to construct the unified sequence? If the order is random then I suspect that the encoder may not be time-aware, making the output less reliable without additional input of time.
The paper is overall well written and the experiment section is clearly expained with newest benchmark methods. Effectively capturing the inter- and intra- series information is an important issue in LTSF, and the idea of unified attention seems interesting.
This paper proposes a novel method to capture inter-series and intra-series information from multivariate time series. This paper utilizes a unified attention mechanism on the flattened patch tokens, and adds a dispatcher module which reduces the complexity and makes the
model feasible for high-dimensional inputs. The model achieves compelling performance in multiple datasets for time series forecasting.
1. The paper claims that 'previous transformer models lack ability to simultaneously capture both inter-variate and intra-variate dependencies', which is not accurate. In fact, there is some literature focusing on modeling inter- and intra- series information in a Transformer structure, for example, CATS: Enhancing Multivariate Time Series Forecasting by Constructing Auxiliary Time Series as Exogenous Variables (ICML 2024) adopts a very similar idea by constructing an auxiliary series to capture inter- and intra- series. The author should revise their literature review accordingly and explain their difference with CATS. 

2. Time series forecast using LLM has been a trand in recent years. The author is encouraged to include more LLM-based methods as benchmarks, such as LLM4TS or GPT4TS.
@inproceedings{
anonymous2024unitst,
title={Uni{TST}: Effectively Modeling Inter-Series and Intra-Series Dependencies for Multivariate Time Series Forecasting},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=cuFnNExmdq},
note={under review}
}
Transformer-based models have emerged as powerful tools for multivariate time series forecasting (MTSF). However, existing Transformer models often fall short of capturing both intricate dependencies across variate and temporal dimensions in MTS data. Some recent models are proposed to separately capture variate and temporal dependencies through either two sequential or parallel attention mechanisms. However, these methods cannot directly and explicitly learn the intricate inter-series and intra-series dependencies. In this work, we first demonstrate that these dependencies are very important as they usually exist in real-world data. To directly model these dependencies, we propose a transformer-based model UniTST containing a unified attention mechanism on the flattened patch tokens. Additionally, we add a dispatcher module which reduces the complexity and makes the model feasible for a potentially large number of variates. Although our proposed model employs a simple architecture, it offers compelling performance as shown in our extensive experiments on several datasets for time series forecasting.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Multivariate Time Series Forecasting
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/420d94609cb44327a07b04f84e2ddaa076697d93.pdf
learning on time series and dynamical systems
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
UniTST: Effectively Modeling Inter-Series and Intra-Series Dependencies for Multivariate Time Series Forecasting
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission