Towards Generalisable Time Series Understanding Across Domains
Time Series Analysis;Multi-Domain;Self-Supervised Learning
Yes
N/A
No ethics review needed.
See details in weakness.
1. This paper targets a very important research problem, the time series foundation model. Because the time series data has very high variance across different domains and different tasks. How to integrate them and train one foundation model remains challenging.
2. This paper has a very high quality of preparation. The writing, the organization, and the figure are prepared nicely and with enough details.
3. The results shown in Tab 1, 2, and 3 are competitive compared to baseline TS models.
This paper presents OTiS, a pre-trained foundation model on large-scale time series data to support multi-tasks across domains. Extensive experiments are conducted to demonstrate the powerful performance of the foundation model. This paper is prepared in high quality and can be accepted.
1. Add a subsection to show which category baselines will be compared. For example, traditional TS model, deep learning, TS foundation model, etc.
2. I expect a comparison with some SOTA TS foundation models. For example, https://arxiv.org/abs/2405.02358 . If this part is added, that would be great.
3. Currently, the author use fine-tune to adapt the pre-trained model to various downstream tasks. Can you also add one more subsection to test the prompting on this TS foundation model? That would be another great point.
Yes
No ethics review needed.
Please feel free to address any misunderstandings I've stated in the weaknesses. Answers to the following questions would help me better calibrate my score:
1. How long does it take to finetune on new tasks?
2. How does the finetuned model perform compared to task-specific models? Are these testing datasets really good cases that need pre-trained models?
3. How do you get the ground truth embeddings in Figure 3?
4. Is there any intuition around what information could be shared across such different domains to make pre-training on them useful?
This paper has many strengths:
* The key idea to condition the model on different variables and domains is good. Indeed many related works effectively ignore this information.
* The paper is overall written quite well and arguments are presented clearly.
* The experiments investigate multiple axes, including ablation of their method and different dataset and model sizes, and visualizations of the embeddings.
* The public data and model weights will help the community build on this work.
This paper proposes a time series model architecture and a pre-training objective. The key idea is that their architecture acknowledges that training and testing time series may have different sampling rates and variables. The authors propose a straightforward tokenization scheme to learn embeddings for different variables, which can get added onto regular patch and temporal embeddings, thereby conditioning the predictions on the measured variables. They then pre-train their model on a collection of existing datasets, and evaluate its performance by finetuning on new datasets for some forecasting, regression, and classification datasets. They find that finetuning their model on new datasets can outperform other recent methods.
This paper has weaknesses to address:
* The major weakness of this paper is the extremely limited experiments section. There are many experiments, yet almost no explanation of how they're run or interpretation of the results. Most of the results are written like an advertisement, mostly just stating the method outperforms others. This leaves the reader unclear why the performance gains happen. Ultimately it's not clear when/why the findings would generalize. The result is that some claims appear to be quite overstated. For example, L423-L424 states *"embeddings of domains with shared high-level semantics cluster together, as depicted in Appendix E.1. For example, embeddings of mono and stereo audio group closely, as do those of banking and economics."* But this is cherry-picked---Temperature is way closer to Mono and Stereo Audio than Banking is to Economics.
* Similarly, many important experimental details are missing or relegated to the Appendix, and the Appendix also includes almost no explanations or interpretations. For example, the PCA experiments in Figures 3, 7, and 8 aren't explained.
* It's unclear how many variables actually overlap between training/testing, which seems to be a key element to make the model outperform others. Yet this isn't analyzed. Showing that others fail by ignoring other variables should be a key element of the experiments.
Yes
Authors use Github Link to share the code, leading to potential personal information leakage. This may require further investigation.
Yes, Privacy, security and safety
- How does the domain-specific tokenizer adapt to unseen domains with distinct variate structures?

- Additionally, how does the domain-specific tokenizer generalize across different systems within the same domain? For instance, while both electrical transformers and power generators belong to the "energy" domain, they exhibit differing properties and produce distinct time series readings. How does the sub-domain adaptation discussed in Section 3.1 address this scenario?

- A broader question, not specific to this paper: At what level of granularity should we define the domain?
- The paper is well-written and easy to follow. 

- Authors spot the important fact that the variate structure is heterogeneous across domains and this structure may represent more complex relationships. 

- The visualization for variate embedding seems to be interesting and insightful. 

- A substantial portion of this research focuses on EEG signals, which presents a novel and promising approach. The authors introduce an innovative method to model a "specific set of systems" that, despite being observed differently—such as TDBrain and SEED with 19 channels versus LEMON with 62 channels—remain comparable.
Authors spot the important fact that the variate structure is heterogeneous across domains and this structure may represent more complex relationships. Thus, they propose a time series pre-training pipeline called OTiS. The OTiS is composed of a specially designed tokenizer that can add domain-specific signature to the time series and a novel loss for pretraining.
- As noted in the strengths, this work addresses the challenge of generalizing across datasets that contain time series of similar systems but are recorded differently, such as variations in sampling rates and physical values. However, the claims regarding cross-domain generalization may be overstated.

- From the perspective of generalized time series analysis, the primary contribution of variate-specific embedding may not be effective in other systems where the interrelationships between variates are not as straightforward as their spatial arrangement (e.g., the electrodes in EEG as depicted in Figure 3 of the manuscript). In different physical systems, two variates may exhibit complex computational relationships (e.g., voltage and current as described by Ohm's Law), complicating the direct modeling of variates as embeddings.
Yes
No ethics review needed.
See the weaknesses
1. The paper is well-written, and the method is easy to understand. The authors clearly articulate how they consider the heterogeneity of different domain time series to achieve multi-domain time series forecasting.

2. This paper focuses on the problem of multi-domain time series analysis, which is crucial for building generalizable foundational models for time series.

3. The experimental section utilizes a large amount of data, and the model is open-source, contributing particular engineering value to the community.
This paper presents OTiS for multi-domain time series analysis, building on existing pre-training paradigms for time series. It allocates domain-specific variable embeddings to distinguish the heterogeneity of different variables across domains and enhances the model's ability to learn temporal causal relationships through a dual-masking strategy. Additionally, it introduces NCC loss to capture global patterns. Experimental results demonstrate that the proposed method achieves competitive performance in time series classification, regression, and forecasting tasks across multiple domains compared to SOTA methods. Visualization results further highlight the effectiveness and interpretability of the domain-specific variable embeddings.
1. The paper mentions that one of the challenges of cross-domain time series models is the significant differences in temporal dynamics and sampling frequencies among different domains. However, the paper uses the same patch size for all domains when dividing patches, failing to accommodate the unique sampling rates of different domains. This oversight means the paper does not sufficiently consider the differences in sampling rates across domains. Additionally, using a shared patch projector to encode the temporal dynamics within each patch does not adequately address the differences in temporal dynamics between domains. While this approach may be common in previous works, it does not consider the temporal heterogeneity among domains.

2. The method of considering variable heterogeneity through learned variable embeddings is not uncommon. In spatiotemporal prediction, some methods [2][3] have already employed learnable embeddings to explicitly distinguish heterogeneous spatiotemporal patterns by learning time-specific and space-specific parameter spaces.

3. [1] proposed using textual descriptions to label different time series domains for cross-domain time series forecasting, utilizing a channel-independent strategy. In contrast, the domain-specific variable embeddings in this paper correspond to a channel-mixing strategy. I look forward to seeing a comparison between these two strategies in cross-domain time series.

4. The experimental section lacks details about the baselines. How were these methods selected? Were they pre-trained and fine-tuned? If so, what data was used for pre-training and fine-tuning?

5. How does the performance of the proposed method compare to conventional time series classification or forecasting methods trained on a single specific dataset?

[1] UniTime: A Language-Empowered Unified Model for Cross-Domain Time Series Forecasting, WWW, 2024

[2] Heterogeneity-Informed Meta-Parameter Learning for Spatiotemporal Time Series Forecasting, KDD, 2024

[3] Adaptive Graph Convolutional Recurrent Network for Traffic Forecasting, NeurIPS, 2020
Yes
No ethics review needed.
1. Have you tried to pre-train separately according to different domains and then fine-tune it for domain-specific downstream tasks? As observed from Table 1, there are several discrepancies in different domains, such as the frequencies of Economics and EEG. Is it possible that separating datasets to pre-train domain-specific models works better?
2. The proposed method uses a fixed context for pre-training. Padding a large pre-training corpus, which generally contains univariate time series, into a fixed temporal/variate dimension. Will it cause a waste of computing resources?
1. This paper researches an important question about generalizable time series understanding across diverse domains.
2. This work presents a large pre-training corpus, which can be greatly beneficial if the datasets are released.
3. The method exhibits promising results in handling multivariate time series analysis by leveraging variate and domain signatures.
The paper presents OTiS, a deep model pre-trained on a large corpus (11B) for general time series analysis. In this paper, the authors highlight the challenge of heterogeneity when applying self-supervised pre-training on time series. An MAE-style pre-training method is adopted to obtain a general tokenizer for multivariate time series, and then different task heads are introduced to complete time series analysis tasks.  The model demonstrates strong performance across 15 diverse applications, including time series classification, regression, and forecasting.
1. My major concern is about the novelty of the proposed method: The design of the encoder/decoder is very identical to MAE. Is there any adaptation for the time series modality? For example, considering the inherent reconstruction difficulties of time series and adjusting the mask ratio compared with the vision modality?
2. About the model design towards generalizable time series understanding: As the authors mention an important challenge of heterogeneity, I am slightly unconvinced that a shared unified patch embedding/projector can reflect different semantics among variates and domains, even if the patch is completely the same. Prior to this, Moirai adopted different patch sizes for different frequencies, will it further enhance OTiS?
3. This work adopts learnable embeddings as variate/domain signatures. I am convinced that the signatures can "distinguish" them, but how can they explicitly "capture inter-variate relationships"? This approach may also limit the generalization scope as the learned signatures do not apply to unseen variates/domains during inference.
4. About the experiments: Results of classification are not compared with supervised, trained deep models, for example, TimesNet and ModernTCN. For the regression rask, can you introduce some variate-centric models into this baseline, such as iTransformer? As for forecasting, the average improvement does not seem significant compared with PatchTST. Also, can you provide some explanations about Table 3 why OTiS has a significant improvement on some datasets (such as ETTh2) and a great degeneration on similar datasets like ETTh1?
5. A minor suggestion: the name "dual masking strategy" can be somewhat overstated to me, which generally refers to dual or antagonistic behavior (e.g., minimax). I would prefer to simplify the contribution as a "mixture" (of masking modeling and generative modeling in this paper), which is a common technique in fact. Also, I would like to know how the ratios (25% - 75% in this paper) of the two strategies are determined.
6. The pipeline of using the masked pre-trained models seems still somewhat tedious, i.e., lacking in generalization. Supervised training should be performed after large-scale pre-training. Can the author provide an overall promotion compared with training from random initialization, or try zero-shot generalization on downstream tasks?
@inproceedings{
anonymous2024towards,
title={Towards Generalisable Time Series Understanding Across Domains},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=39n570rxyO},
note={under review}
}
In natural language processing and computer vision, self-supervised pre-training on large datasets unlocks foundational model capabilities across domains and tasks. However, this potential has not yet been realised in time series analysis, where existing methods disregard the heterogeneous nature of time series characteristics. Time series are prevalent in many domains, including medicine, engineering, natural sciences, and finance, but their characteristics vary significantly in terms of variate count, inter-variate relationships, temporal dynamics, and sampling frequency. This inherent heterogeneity across domains prevents effective pre-training on large time series corpora. To address this issue, we introduce OTiS, an open model for general time series analysis, that has been specifically designed to handle multi-domain heterogeneity. We propose a novel pre-training paradigm including a tokeniser with learnable domain-specific signatures, a dual masking strategy to capture temporal causality, and a normalised cross-correlation loss to model long-range dependencies. Our model is pre-trained on a large corpus of 640,187 samples and 11 billion time points spanning 8 distinct domains, enabling it to analyse time series from any (unseen) domain. In comprehensive experiments across 15 diverse applications - including classification, regression, and forecasting - OTiS showcases its ability to accurately capture domain-specific data characteristics and demonstrates its competitiveness against state-of-the-art baselines. Our code and pre-trained weights are publicly available at \url{https://github.com/OTiS-official/OTiS}.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Time Series Analysis Multi-Domain Self-Supervised Learning
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/e4d2cf724e8953bd4834c764d48045f0827da18e.pdf
learning on time series and dynamical systems
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
/attachment/662380416ac92716a8d258d31a6b8c8e293e316d.zip
Towards Generalisable Time Series Understanding Across Domains
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission