T-Graphormer: Using Transformers for Spatiotemporal Forecasting
Deep Learning;Spatiotemporal Forecasting;Transformer;Graph Neural Network
Yes
No ethics review needed.
Please refer to the weaknesses.
Two novel spatial feature extraction methods are proposed in this paper, one for encoding the in-degree and out-degree of the graph structure and the other for encoding the shortest travel distance between graph nodes.
The manuscript focuses on spatiotemporal forecasting. To simultaneously learn both spatial and temporal correlations, the authors propose T-Graphormer based on the transformer architecture and design temporal and spatial encoding methods for this purpose.
1. There have been several methods [1][2] that apply transformer architecture to spatio-temporal forecasting. The idea of using transformers to simultaneously capture temporal and spatial correlations is not sufficiently novel.
2. Section 4.3 does not adequately explain the setting of additional tokens. These tokens are typically used in the NLP domain, but the authors fail to clarify their significance in spatio-temporal forecasting or provide details on their implementation.
3. The experimental design has shortcomings. First, the dataset selection is insufficient. The study focuses on spatio-temporal forecasting but only uses two traffic speed datasets to validate the model, which is inadequate to demonstrate the method’s superiority. Second, the experimental baseline lacks comparisons with advanced transformer-based methods [1][2]. Finally, the hyperparameter experiments do not sufficiently explore hyperparameter values.
4. The manuscript's structure is also not well-balanced. Chapter 4, which describes the main method, occupies only one and a half pages, while other sections take up considerably more space.
5. The language presentation is rough. The adjacency matrix dimension in Section 3.1 is incorrect. Section 4.1 consists of only two sentences and lacks a period. Additionally, the summary in Section 6 is confusing and poorly presented.

Reference:
[1] Liu H, Dong Z, Jiang R, et al. Spatio-temporal adaptive embedding makes vanilla transformer sota for traffic forecasting[C]//Proceedings of the 32nd ACM international conference on information and knowledge management. 2023: 4125-4129.
[2] Jiang J, Han C, Zhao W X, et al. Pdformer: Propagation delay-aware dynamic long-range transformer for traffic flow prediction[C]//Proceedings of the AAAI conference on artificial intelligence. 2023, 37(4): 4365-4373.
Yes
No ethics review needed.
See Weaknesses.
S1. The author has effectively enhanced Graphormer for spatiotemporal prediction, incorporating improvements in both embeddings and tokens.

S2. T-Graphormer demonstrates superior performance, particularly when forecasting the next 12 time steps (1-hour window).
In this paper, the authors propose an approach called Temporal Graphormer(T-Graphormer). It can models spatiotemporal correlations directly by capturing spatiotemporal correlations and adding special tokens.
Weaknesses:

W1.Key points lack details. While the existing content and formulas in Graphormer are well-articulated, The description of the novel contributions is not sufficiently detailed. In Section 4.2, the paper only presents the modified transformer method for T-Graphormer only in brief with the expression l=T'*N. In Section 4.3, the paper doesn't clearly convey the formulas for special tokens. Could you provide detailed mathematical formulas and subsequent explanations for these original contributions?

W2. Lack of recent related work. The related work mentioned in Chapter 2, Related Works, basically references studies prior to 2020. There's also a lack of references to newer traffic prediction models. Including recent works such as  STAEformer [1]  would strengthen the baselines.

[1]Liu H, Dong Z, Jiang R, et al. Spatio-temporal adaptive embedding makes vanilla transformer sota for traffic forecasting[C]//Proceedings of the 32nd ACM international conference on information and knowledge management. 2023: 4125-4129.

W3. Lack of visualization. While the main claim of the paper is that directly modeling spatiotemporal correlations is better than modeling them separately, there is no accompanying visualization or case study to substantiate this claim, reducing the argument's credibility.

Could you provide a visualization of an attention example for a particular sensor to illustrate that the T-Graphormer indeed learns spatiotemporal patterns effectively compared to previous methods?

Minor remarks:

* In the last paragraph of 4.2, the symbol upright phi appears without prior definition.
Yes
No ethics review needed.
1. See weaknesses.
2. Why you choose STEP as the baseline? It seems that your model does not have a higher performance.
3. Since your title is spatial temporal forecasting, do you agree that more spatial temporal dataset should be concluded, such as weather dataset and crime dataset?
1. Well written related work.
2. Nice figure 2. It clearly explains the model framework.
3. The ablation study is somehow strong and it shows that different model components are useful.
In this paper, the authors develop a spatial temporal forecasting/traffic prediction task with transformer-based structure. Unlike other methods, T-Graphformer can capture spatial-temporal correlations directly and it has a better performance than some of the baseline methods.
1. The baseline methods are outdated. Authors need to compared this model with more advanced and latest baselines, instead of SVM and FC-LSTM. Additionally, more datasets should be included. For example, you might include STID, MegaCRN and PDFormer. 
References:
Zezhi Shao, Zhao Zhang, Fei Wang, Wei Wei, and Yongjun Xu. 2022. Spatialtemporal identity: A simple yet effective baseline for multivariate time series forecasting. In Proceedings of the 31st ACM International Conference on Information
& Knowledge Management. 4454–4458.
Renhe Jiang, Zhaonan Wang, Jiawei Yong, Puneet Jeph, Quanjun Chen, Yasumasa Kobayashi, Xuan Song, Shintaro Fukushima, and Toyotaro Suzumura. 2023. Spatio-temporal meta-graph learning for traffic forecasting. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 37. 8078–8086.
Jiawei Jiang, Chengkai Han, Wayne Xin Zhao, and Jingyuan Wang. 2023. PDFormer: Propagation Delay-aware Dynamic Long-range Transformer for Traffic Flow Prediction. In AAAI. AAAI Press.
For datasets, more traffic datasets, such as PEMS03, PEMS04, PEMS07 and PEMS08 can be included. It is not sufficient to only use two datasets to demonstrate the effectiveness of your model. 
2. The writing should be improved. The authors use 'spatial temporal forecasting', 'multivariate time series forecasting', 'traffic prediction' in the introduction and section. Although these terms are correlated, they are different and they might be better to focus on one specific task while writing this paper. Since your data is traffic, it is better to frame it as a traffic prediction problem and emphasize it in your introduction. 
3. The methodology itself is not innovative. For example, authors do not make modifications to the traditional transformer structure to make improvement. Many papers adjust the attention mechanism to better capture temporal/spatial correlations. Why you only use the original transformer?
Yes
No ethics review needed.
1. The encoding strategy in the methodology seems primarily focused on a single data dimension. However, spatial-temporal data inherently involves complex interactions between spatial and temporal dimensions. Given that the embedding preprocessing is focused solely on the temporal dimension, how could the current method effectively  capture dependencies across both dimensions?
2. The attention mechanism is indeed effective at capturing correlations, but have you considered testing the encoding strategy with alternative mechanisms, such as replacing attention with a Graph Neural Network (GNN), in ablation studies? This would help address the lack of sufficient Transformer or attention-based baselines for comparison in the experiments.
3. Could you provide a brief summary of how the graph was generated and the adjacency matrix constructed? While I understand the implementation follows another paper, it would be helpful if these details were mentioned directly in your paper for clarity.
1. The time series forecasting problem is a valuable and impactful area of research, with significant applications across various fields.
2. This paper introduces a centrality encoding and positional encoding combination approach, with the attention mechanism, to capture complex spatial-temporal correlations. This approach can be easily adapted to various domains.
3. This paper provides a clear presentation and thorough description of the methodology and experimental setup.
The paper presents the Temporal-Graphormer (T-Graphormer) framework, which encodes spatial-temporal data in the temporal dimension and employs the attention mechanism to effectively capture complex spatial and temporal relationships. Comprehensive experiments also demonstrate its effectiveness.
1. This paper claims that T-Graphormer performs well on spatial-temporal data; however, only traffic datasets were used in the experiments, so testing on additional datasets is encouraged. Additionally, the experiments focus primarily on main results and ablation studies, and further experiments, such as hyperparameter analysis, are recommended.
2. This paper would benefit from incorporating more recent citations on state-of-the-art Transformer research, particularly in the baseline comparisons. Adding newer relevant works, such as DSformer [Yu, 2023] and LDT [Feng, 2024], is recommended.
3. Even though detailed information about the implementation are provided, no code provided would still make it hard to evaluate the results.

Reference:
[1] Feng, S., Miao, C., Zhang, Z., & Zhao, P. (2024). Latent Diffusion Transformer for Probabilistic Time Series Forecasting. Proceedings of the AAAI Conference on Artificial Intelligence, 38(11), 11979-11987. https://doi.org/10.1609/aaai.v38i11.29085.
[2] Yu, Chengqing, et al. "Dsformer: A double sampling transformer for multivariate time series long-term prediction." Proceedings of the 32nd ACM international conference on information and knowledge management. 2023.
@inproceedings{
anonymous2024tgraphormer,
title={T-Graphormer: Using Transformers for Spatiotemporal Forecasting},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=m30uro534c},
note={under review}
}
Time series data is ubiquitous and appears in all fields of study. In multivariate time series, observations are interconnected both temporally and across components. For instance, in traffic flow analysis, traffic speeds at different intersections exhibit complex spatiotemporal correlations. This dual structure presents unique challenges for modelling. Most existing forecasting methods address this by learning the spatial and temporal dependencies separately. Here, we propose Temporal Graphormer (T-Graphormer), a transformer-based method that models spatiotemporal correlations directly. By extending the Graphormer architecture over time, each node is updated based on all other nodes within the historical context window, allowing the model to learn powerful representations. We demonstrate the efficacy of T-Graphormer by evaluating it on two real-world traffic prediction benchmarking datasets. Compared to state-of-the-art methods, our method shows a reduction in root mean squared error (RMSE) by up to 10\% and mean absolute percentage error (MAPE) by up to 10\%.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Deep Learning Spatiotemporal Forecasting Transformer Graph Neural Network
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/2d77a019971d6afc0a7b9e961bf9bbb15168788e.pdf
learning on time series and dynamical systems
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
T-Graphormer: Using Transformers for Spatiotemporal Forecasting
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission