Sequential Order-Robust Mamba for Time Series Forecasting
Mamba;State-space Model;Time Series
Yes
No ethics review needed.
* When randomly shuffling the channel order, how is it done? Is a fixed shuffling method used for all sequences in a dataset, or is it shuffled randomly each time?
* In the last row of Table 1, is the calculation formula written incorrectly?
* This paper uses a single unidirectional Mamba with regularization and non-1D convolutions to capture channel correlations. By modifying the traditional Mamba module, it can better address the issue of sequential order bias. The research problem is clearly defined, and the solution is reasonable. And the pretraining task channel correlation modeling also enhances the model's generalization and performance.

* The details of the model are mostly clear, with each module explained and supported by equations. The experiments  include time series forecasting, transfer learning and many ablation studies to validate the effectiveness of each module.
This paper proposes SOR-Mamba to solve the sequential order bias introduced by mamba capturing channel dependencies. Model SOR-Mamba incorporates a regularization strategy to minimize the distance between embeddings from data and reversed data and eliminates the 1D-convolution in the original Mamba block. And a pretraining task channel correlation modeling is introduced to preserve the channel correlation from the data space to the latent space. The effectiveness of the proposed method is demonstrated by the extensive experiment results.
* The proposed method is mostly constructed on existing models -- reverse modeling has been widely used in literature and removing the 1d-conv is trival.
* The experimental results show that the model's performance improvement across various datasets is not significant, mostly in the thousandth.
* The removal of 1D-conv negatively impacts PEMS dataset in tale 7 and figure 5. The necessity to remove 1D-conv is uncertain.
* The model uses the Mamba backbone, but the baseline only includes a single Mamba model: S-Mamba. Providing more models from the Mamba framework for comparison would be better.
Yes
No ethics review needed.
1. In the Manba architecture, are there other advances in order-invariant architectures that can be compared?
2. The definition of sequential order bias in Figure 4 and how it is calculated?
1. Clear problem identification regarding Mamba's limitations in handling unordered channels
2. Comprehensive experiments
3. Improved efficiency compared to other approaches
This paper introduces SOR-Mamba, a time series forecasting method that addresses the sequential order bias  while capture channel dependencies in Mamba, through a regularization strategy to minimize the discrepancy between two embedding vectors generated from data with reversed channel orders, and 1D-convolution removal, The authors also propose Channel Correlation Modeling (CCM) as a pretraining task.
**Limited Technical Novelty**:
- The regularization strategy is overly simplistic that minimizing the distance between embeddings from different channel orders
- The removal of 1D-conv lacks theoretical justification and appears to be an ad-hoc solution, As shown in Figure 5 of the paper, its removal may negatively impact datasets with ordered channels such as PEMS datasets
Yes
No ethics review needed.
1. How does the choice of the distance metric d in the regularization term impact the model's performance across datasets with varying levels of channel correlation? - is the regularization term’s effectiveness dependent on the number of channels or dataset-specific factors etc.?

2. The paper uses CCM to preserve channel correlations in the latent space during pretraining. How does CCM compare to masked modelling and reconstruction pretraining across different dataset sizes and channel configurations? - and is there any limitation of CCM if there's only exist weak channel correlations?
The proposed method addresses the sequential order bias in channel dependencies (CD). This bias can degrade performance when channels lack inherent order in multi-channel time series data. The SOR-Mamba method mitigates this bias by introducing a regularization technique that minimizes the discrepancy between embeddings generated from reversed channel orders, thus improving robustness. 

Additionally, the paper proposes Channel Correlation Modeling (CCM) as a pretraining task, which preserves channel correlations from data to latent space, enhancing CD capture. The architecture modifications, including removing the 1D-convolution and focusing on unidirectional Mamba, can reduce computational overhead, achieving efficiency gains in memory and parameter usage compared to other models like S-Mamba. This should be of interests to the group of readers working on multi channel time series data forecasting.
This paper addresses a limitation in applying Mamba to time series forecasting: the sequential order bias when modelling channel dependencies, as channels typically lack inherent order. The authors propose SOR-Mamba, which introduces two main innovations: a regularization strategy that minimizes discrepancy between embedding vectors from reversed channel orders, and the removal of the unnecessary 1D-convolution layer. They also introduce Channel Correlation Modelling (CCM), a pretraining task that preserves channel correlations from data space to latent space. The method reported promising performance across different scenarios, including cases with missing data and varying channel orders, while maintaining computational efficiency in terms of memory and runtime.
The paper’s approach is relatively complex, involving several architectural changes (e.g., removing 1D-convolutions, applying specific regularization, and adding CCM pretraining) that may complicate replication and limit accessibility. 

While it emphasizes efficiency improvements, the paper could provide more detailed explanations regarding the trade-offs involved in removing the 1D-convolution, especially on datasets where channel order may have some inherent structure (e.g., traffic data). The study also lacks a comprehensive discussion of potential limitations in real-world deployment, such as sensitivity to hyperparameters (e.g., λ in regularization) and its impact across different datasets.
Yes
No ethics review needed.
1. Table I.1 seems to show that the hyperparameter lambda does not affect the results. Is this a reasonable phenomenon for the key contribution?
2. What are the principles and motivations for using parameter-sharing CD-Mamba block and removing 1D-conv? Can the authors provide additional details?
3. The description of the pre-training method used in this study is somewhat vague. Can the authors supplement the algorithm flow or provide detailed explanation?
1. This study achieves a good improvement in results with only a few changes to the method. The proposed method is simple, novel and effective.
2. The experiments carried out by the authors are very detailed and the relevant parameters are well discussed. This is an experimentally rigorous and solid study.
3. The manuscript is well-written and the figures are clear and intuitive.
This manuscript proposes an algorithm to regularize bidirectional Mamba channel vectors in order to solve the problem that there is no intrinsic order relationship between channels in time series data. The authors also design a new pre-training method (CCM) to help the model better establish the association between channels of multi-dimensional time series. Extensive experiments and ablation experiments are conducted, and the experimental results show the effectiveness of the relevant contributions.
1. The authors lack a complete description of the motivation for regularization. Why regularizing the bidirectional mamba channel vectors and minimizing the distance between them can help the model better capture the correlation between channels? This requires the author to give a more intuitive and reasonable motivation.
2. Although the authors have done a lot of experiments, they lack experiments on longer input lengths. Some widely used baselines in this field, such as PatchTST (arXiv preprint arXiv:2211.14730, 2022.), all use longer input sequences.
3. The authors use too many abbreviations in their manuscripts and experimental result tables, and lack clear and complete explanations of the abbreviations. This makes the relevant content very difficult to read and easily confusing.
Yes
I think these if no ethics concerns.
No ethics review needed.
See Weaknesses above.
3. In line 369-370, I’m curious about how to calculate the global correlation.
1. The proposed SOR-Mamba uses a modified Mamba structure (also with a regularization strategy) to eliminate the sequential order bias of Mamba which is more useful in capture channel dependencies.
2. The authors also introduce a pretraining task named channel correlation modeling (CCM) to preserve correlations between channels from the data space to the latent space in order to enhance the ability to capture CD.
This paper introduce a new Mamba-based model for long term time series forecasting named SOR-Mamba which utilizes modified Mamba structure to capture channel dependencies. Furthermore, the authors also introduces channel correlation modeling as a pretraining task aimed at enhancing the ability to capture channel dependencies.
1. Since Mamba is a sequential-based model, and the channels lake sequential order, why we need Mamba to extract channel dependencies? It is important to clarify the importance of capturing channel dependencies with Mamba.
2. The experimental comparisons appear unfair, as models like PatchTST exhibit better performance with longer lookback window. I strongly suggest the authors conduct additional experiments, such as setting $L=336$.
Yes
N/A
No ethics review needed.
Please see weaknesses.
1. The authors propose SOR-Mamba, a TS forecasting method that handles the sequential order bias by
1) regularizing the unidirectional Mamba to minimize the distance between two embedding vectors
generated from data with reversed channel orders for robustness to channel order.
2. The authors introduce CCM, a novel pretraining task that preserves the correlation between channels from
the data space to the latent space, thereby enhancing the model’s ability to capture CD.
3. With the exception of the power-related datasets (ETTm1 & m2, etc.), the experimental performance exceeds the current SOTA.
In this manuscript, the authors claim that channels in time series (TS) data have no specific order in general, recent studies have adopted
Mamba to capture channel dependencies (CD) in TS, introducing a sequential order bias. To address this issue, the authors propose SOR-Mamba, a TS forecasting method that 1) incorporates a regularization strategy to minimize the discrepancy between
two embedding vectors generated from data with reversed channel orders, thereby enhancing robustness to channel order.
1. It is puzzling why the authors mention 'robust' but does not give experiments to resist noise or PGD attacks. Or it may be necessary to modify the terminology in the manuscript to be more precise, after all, 13 different domains of time series prediction tasks have been tried.
2. With the exception of the power-related datasets (ETTm1 & m2, etc.), the experimental performance exceeds the current SOTA. Why is the performance of the author's scheme inferior to SOTA (i.e., PatchTST) on power-related datasets (ETTm1 & m2, etc.) with more significant global regularity?
3. Despite the computational complexity comparison, the reviewer is concerned about the actual training and inference time.
4. The description in Figure 1 seems to be the difference between different exogenous time series due to sampling frequency and periodic change rule, and the description of sequence order is prone to misunderstanding (i.e., covariables have orders).
We propose SOR-Mamba, a time series forecasting method that uses Mamba with regularization to address the sequential order bias in capturing channel dependencies.
@inproceedings{
anonymous2024sequential,
title={Sequential Order-Robust Mamba for Time Series Forecasting},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=fyl82vAale},
note={under review}
}
Mamba has recently emerged as a promising alternative to Transformers, offering near-linear complexity in processing sequential data.
However, while channels in time series (TS) data have no specific order in general, recent studies have adopted Mamba to capture channel dependencies (CD) in TS, introducing a sequential order bias. To address this issue, we propose SOR-Mamba, a TS forecasting method that 1) incorporates a regularization strategy to minimize the discrepancy between two embedding vectors generated from data with reversed channel orders, thereby enhancing robustness to channel order, and 2) eliminates the 1D-convolution originally designed to capture local information in sequential data. Furthermore, we introduce channel correlation modeling (CCM), a pretraining task aimed at preserving correlations between channels from the data space to the latent space in order to enhance the ability to capture CD.
Extensive experiments demonstrate the efficacy of the proposed method across standard and transfer learning scenarios.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Mamba State-space Model Time Series
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/4be412bfd2514287b7d6c41fe8d30b5368946543.pdf
learning on time series and dynamical systems
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
/attachment/a3e6bf94e3ea1b0642f98af3f08c4590c0c5d03e.zip
Sequential Order-Robust Mamba for Time Series Forecasting
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission