TimeDiT: General-purpose Diffusion Transformers for Time Series Foundation Model
Time Series; Foundation model; Diffusion model
Yes
No ethics review needed.
1. The scales of the model and training data are crucial to the performance and generalizability of foundation model. Can the author provide a summary of the datasets used in pre-training and fine-tuning and the scale of models in the number of parameters? A good reference could be Table 2 in this survey work[1] of foundation models for time-series. This detail would be critical to assess the contribution of the work.

2. Incorporating external physics knowledge would require having knowledge of the partial differential equation (PDE) that characterizes the underlying dynamics. The problem setting and practical values of the approach itself require further justification. Since we already have access to the ground truth dynamics, can we directly generate training samples by solving the PDE numerically and training a separate model? If it is feasible, how does it compare to the Physics-Informed TimeDiT in the work?

References

[1] Ye, Jiexia, et al. “A Survey of Time Series Foundation Models: Generalizing Time Series Representation with Large Language Mode.” arXiv preprint arXiv:2405.02358 (2024).
The work exhibits the following notable strengths:
1. The work effectively demonstrated the value of integrating masked reconstruction with diffusion models in the domain of time-series data.
2. Extensive experiments are conducted to illustrate the potential of this approach as a foundational model for the proposed methodology.
3. The incorporation of PDE-based external physics knowledge into diffusion models for time-series is a novel and intriguing concept.
The proposed work presents a method for training a diffusion model for time-series based on masked reconstruction. This model models the distribution of masked time series values conditioned on the observed values. Various masking strategies with different inductive biases are designed for pre-training and task-specific fine-tuning. Additionally, external physics knowledge on time-series dynamics in the form of partial differential equations (PDEs) can be incorporated into samples generated by the model without re-training the model through changes to the sampling dynamics. The proposed model demonstrates impressive performance across various time-series tasks, including imputation, forecasting, and anomaly detection. Furthermore, it exhibits generalizability across datasets in zero-shot forecasting. The experimental results suggest the potential of a foundation model for time-series data.
The work presents several notable weaknesses:

1. The lack of detailed information regarding the datasets utilized for pre-training and fine-tuning the model poses a significant concern. If each task necessitates a specific subset of datasets for pre-training, the work risks over-claiming its role as a foundational model for time-series.

2. The primary methodology contribution of the work, which combines masked reconstruction with diffusion models, lacks novelty as similar ideas have been explored in the image domain ([1, 2]).

3. Sequence-level classification is a crucial task in time-series modeling ([3, 4]). Consequently, a foundational model for time-series should demonstrate its capability in handling this task. Notably, this aspect is absent from the experimental results presented in the work.

References

[1] Wei, Chen, et al. “Diffusion models as masked autoencoders.” Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.

[2] Gao, Shanghua, et al. “Masked diffusion transformer is a strong image synthesizer.” Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.

[3] Kidger, Patrick, et al. “Neural controlled differential equations for irregular time series.” Advances in Neural Information Processing Systems 33 (2020): 6696-6707.

[4] Morrill, James, et al. “Neural rough differential equations for long time series.” International Conference on Machine Learning. PMLR, 2021.
Yes
No ethics review needed.
* How sensitive is model performance to the choice of mask ratios during pretraining?
* Could you please provide further information about how TimeDiT processes inputs with different channel dimensions?
* With physics-informed sampling, could you comment on how one manages the tension between the demands introduced by physical constraints on the one hand and learned distributions on the other?
* Concerning the adaptive layer normalization AdaLN, how does it stack up compared to other conditioning methods one may have tried?
* Which was the rational for choosing those very specific transformer architecture details, e.g., number of heads and layer depth?
* For the forecasting experiments that involve missing values, how were the missing patterns generated? Are results robust against different missing data mechanisms, MCAR, MAR, MNAR?
* In the anomaly detection task, how sensitive are the results against the choice of 99th percentile threshold shown, did you experiment with others?
* How do you control that the generated samples in synthetic generation experiments are temporally coherent?
* Which part of the architecture is most responsible for the performance improvement over baselines?
* When TimeDiT is not outperforming baselines, what general features does such a setting have?
* Please provide more intuition about Theorem 4.1 and its practical implication.
* How does the choice of the diffusion schedule influence the incorporation of physical constraints?
* Which methods were explored for stabilizing the training of this combined diffusion model and transformer?
* How was the trade-off done between sample quality and the number of sampling steps?
* Techniques to integrate diffusion models and transformers for time series
* Extensive experimental validation on a variety of tasks and datasets
* Usability in real-world along with the ability to address the actual challenges
* Theoretical properness with physics-informed sampling
This paper introduces TimeDiT, a general-purpose foundation model for time series analysis, which leverages transformer architecture with the diffusion model.

* A unified masking strategy that allows the handling of multiple tasks, such as forecasting, imputation, and anomaly detection.
* Direct handling of missing values and multi-resolution data using adaptive mechanisms in input processing.
* Integration of physical knowledge into the sampling via an energy-based prior
* It provides state-of-the-art performance on various benchmarks and practical scenarios.
* Limited discussion about the computational costs of the models, and efficiency in training.
* More could be done by way of analysis on model scalability for very long sequences.
* Limitations Relatively short discussion of
* Providing deeper analysis about failure cases would be good.
Yes
No ethics review needed.
See Weaknesses
- Overall, the work has strong experimental results.
- This work introduces new method to perform cross-dataset training on multiple tasks.
In this work, the authors introduce Time Diffusion Transformer (TimeDiT), a foundational model for time-series data that leverages a diffusion transformer architecture. The authors also propose a novel approach for integrating physical knowledge into foundational time-series models, enhancing their applicability to real-world scenarios. Extensive experiments are conducted across multiple tasks—including forecasting, imputation, and anomaly detection—demonstrating the model's potential effectiveness across a variety of applications
The following aspects of the paper require clarification and improvement to enhance its reproducibility, understanding, and relevance.

### Methodological Clarity and Missing Technical Details

Several essential technical details are missing in the Method section, making it challenging to fully understand the proposed approach:

- Token Definition: The authors should specify what is considered a "token" in the model, as this definition is critical for understanding the data representation.
- Embedding Architecture: The architecture used to encode each time point $x_i$ (referred to as the "embedder") is unclear. For example, is it a simple linear projection, a convolutional layer, or another design? The appendix also lacks implementation details about this embedder. Given that the paper proposes a new architecture, the engineering choices for this component should be discussed.
- Noise Embedding Justification: The rationale behind embedding the noised sample instead of the original observation is not clearly explained. It would be valuable to discuss the intuition behind embedding noise, as opposed to embedding the original observation and applying the diffusion process in the embedded space. This additional context could help readers understand the reasoning and potential advantages of this approach.

### Irregular Time-Series Data Claims and Evaluation
- In the introduction, the authors claim that their method can handle irregular time-series data, supported by real-world examples. However, the experimental section only evaluates the model on an imputation task, which does not fully align with the challenges posed by irregular time-series data. In irregular time-series modeling, the input often contains missing data by default, and the model learns from partially observed inputs, as demonstrated in related works like [1] and [2].

- To substantiate the claim that TimeDiT can handle irregular time-series, it would be beneficial to include experiments that specifically address irregular data scenarios, such as handling incomplete observations in generation tasks.

### Comparison with Established Benchmarks
The paper does not include a comparison with well-established benchmarks for time-series forecasting, such as the one proposed in [3]. Including such a comparison would allow readers to better understand how the proposed method performs relative to existing approaches in the field. This is particularly important given the paper’s goal to demonstrate generalizability and robustness across various time-series tasks.

### Adaptability to Different Channel Numbers and Input Resolutions
The adaptability of the proposed model to inputs of different resolutions and channel numbers is not clearly addressed.

Channel Adaptation: Does the model accommodate varying channel counts via padding or another method? The paper suggests a predefined maximum resolution for signal input, but it is unclear how the model would handle cases where the input resolution might exceeds this maximum, such as in a zero-shot setting.
It would be helpful if the authors could elaborate on the selection criteria for the maximum resolution. How was this limit determined?



*[1] "GT-GAN: General Purpose Time Series Synthesis with Generative Adversarial Networks", Jinsung Jeon et al.*

*[2] "Generative Modeling of Regular and Irregular Time Series Data via Koopman VAEs", Ilan Naiman et al.*

*[3] "One Fits All:Power General Time Series Analysis by Pretrained LM", Tian Zhou et al.*
Yes
No ethics review needed.
**Question 1:** 

We argue that TimeDiT simply couples multiple tasks within a single framework via a masking paradigm, and does not substantially address differences in the level of inductive bias across tasks (see weaknesses for details). We consider this paper as incremental work rather than an innovative contribution. 

Indeed, if we splice a trainable cls-token after the sequence as an identifier for the classification task, can we claim to have proposed a base model that takes into account both classification and prediction tasks?

**Question 2:** 

Consider that the random masking strategy in the pre-training phase destroys the coherence of the time series data, which leads to the loss of information such as trend-season, multi-resolution, and multi-periods. Recent research (SimMTM[8]) has shown that pre-training a foundation model with sufficient cross-domain generalization based on the masked self-supervision strategy often requires a large amount of unlabeled data and a slow learning process. 

Furthermore, fine-scale and coarse-scale supervision usually contain low-level and high-level information from the observation, respectively, and TimeDiT only applies to single-scale supervision for reconstruction, widely ignoring multiscale supervision information from the inputs, how can TimeDiT overcome these challenges?

**Question 3:** 

Although the authors claim that TimeDiT "enables direct processing of multivariate inputs", the methodology lacks a design that establishes cross-channel connectivity.   Specifically, instead of designing novel structures to replace the Channel Independent and Patch Embedding designs, TimeDiT still adopts the primitive ‘each moment as a token’ strategy, yet PatchTST and a series of models (OneFitsAll, TimeLLM, Timer, Moment) based on the ‘Patch Embedding’ design have achieved significant success.  

According to existing research, a single Time point has limited temporal information, so this naive "Time as Token" strategy has been replaced by "Patch as Token" strategy.  The authors are expected to design more ablation experiments to verify the superiority of the proposed modelling approach over the Channel Independent and Patch Embedding strategies.

**Question 4:** 

See Weakness for our concerns about the experimental results, and we believe that the introduction of competitive and up-to-date baselines is considered necessary.

**Question 5:** 

The results of Baseline presented in Table 6 are inconsistent with the results presented in the original paper, e.g., the Discriminative Score of the Diffusion-TS model under the Stocks dataset in Table 6 is 0.1869, whereas it is reported as 0.067 in the original paper. In fact, **the results of all the Baselines in Table 6 have a large discrepancy**. The authors must give further experimental details to explain the potential reasons for these deviations.

**Question 6:** 

In Section 5.7, the authors introduced frequency and category information to align textual and temporal data, which significantly improves the zero-sample capability of TimeDiT. However multimodal alignment is a complex and complete endeavor, has TimeDiT designed a unique structure to accommodate the introduction of textual information? Does it utilize pre-trained LLMs to help align text and temporal modalities? The fact that aligning textual and temporal information to guide the noise reduction process of the diffusion model does not yet exist, we think this is an interesting contribution, however unfortunately **the description of Multi-model TimeDiT is missing** in both the main text and the appendices, and we look forward to the authors discussing the implementation details of this part further.

**Question 7:** 

The authors claim to have proposed a generalized time series model based on a self-supervised paradigm, however, the main paper and appendix sections **lack a detailed description of a "unified pre-trained temporal dataset"**. We hope that the authors can give some guidance, such as how to combine a large number of small datasets (19 datasets are included in the Appendix section) into a general dataset, and in particular how the authors can deal with the fact that cross-domain datasets tend to have different channel counts, sampling rates, and data sizes. 

The established temporal basis models generally agree that establishing a unified dataset has a large impact on model performance, and it is expected that the authors will discuss more details of the experimental design.

**Question 8:** 

An inescapable and confusing problem is the seeming **lack of ablation experiments** in the main paper and appendix sections, which makes it difficult to judge the effectiveness of the individual components in TimeDiT. In fact, the authors claim to have introduced a number of unique designs in TimeDiT, including 1) Transformer-based Condition Injection; 2) Time Series Mask Unit; and 3) Physics-Informed. Adequate ablation experiments are considered important in order to facilitate the reader to further understand the impact of each component on the overall performance.

**Reference:** 

8) Dong, Jiaxiang et al. “SimMTM: A Simple Pre-Training Framework for Masked Time-Series Modeling.” NIPS 2023.
This paper discusses the potential challenges that may be faced in establishing a generic diffusion-based time series model, such as variable channel sizes across domains, missing values, and multi-resolution characteristics prevalent in the real world. Furthermore, the ideal foundation model is expected to be simultaneously compatible with a wide range of downstream tasks in the time series domain, such as forecasting, imputation, anomaly detection and classification. Due to their unique ability to capture uncertainty and randomness from time series domain, diffusion models are seen as promising theories for establishing foundation models. In addition, exploiting multimodal textual information in order to improve the performance of foundation models in zero-shot forecasting is also seen as promising work.
This paper proposes TimeDiT, a generalized model for time series based on the diffusion, where masking units are designed for task-independent pre-training and task-specific sampling, and allow for the flexible integration of external knowledge during the sampling process in order to create models that can seamlessly deal with the diversity and complexity of time series data with varying lengths of history and characteristics. In addition, TimeDiT incorporates physical knowledge to further improve model performance and interpretability, and utilizes the Transformer's inductive bias to capture time-dependent and diffusive processes to generate high-quality candidate samples.
**Weakness 1:** 

According to the authors, TimeDiT "can unify diverse time series tasks within a single generative framework". Since imputation and anomaly detection tasks are implemented utilizing the same mask-reconstruction strategy, TimeDiT is essentially self-supervised pre-training first, and then sampling on specific tasks without fine-tuning.

However, there are differences between masking and forecasting: 

* the masking task instructs the model to capture contextual information from multiple randomly masked and discontinuous subsequences for reconstructing the masked portion. 

* the forecasting task requires the model to capture global dependencies and local fluctuation patterns, such as seasonal-trend, multi-scale, and multi-period information, from the observed sequences, which are often corrupted by random masking operations.

* the reconstruction paradigm allows the use of local supra- and infra- information to generate the masked portion, whereas the prediction paradigm allows the use of observed sequences only to predict future sequences.

These differences make it difficult for the temporal representations learned in the masking task to be used directly to model observed sequences.

**Weakness 2:** 

TimeDiT's Transformer backbone still adopts the traditional Encoder-only architecture consisting of Input Embedding Layer, Multi-layer Encoder, Output Projection Layer, which is in line with most of the existing articles (PatchTST, iTransformer, UniTime, Timer, Moment, Moirai) and thus lacks innovation. Moreover, this architecture is unable to cope with the inherent challenges of temporal data, such as multi-scale and multi-period temporal characterization.

**Weakness 3:** 

In the denoising process, the conditional information captured from the observation sequences is directly spliced into the input sequences, however, recent studies (ControlNet[1]) have shown that such a simple scheme cannot guarantee a high multivariate temporal correlation between past observations and future predictions. 

Indeed, many diffusion prediction methods (mr-Diff[2], MG-TSD[3]) are dedicated to efficient architectural design on conditionally denoised networks. Introducing consistency complementation in the progressively denoised diffusion training process will help to improve the generation accuracy and generalization ability of diffusion models for tasks such as time series prediction and interpolation. 

Specifically, if the observation sequence contains multi-resolution features, an intuitive design is to construct a multi-level representation extractor and subsequently reconstruct coarse- and fine-grained fluctuation patterns successively during the denoising process. 
In summary, we argue that the plain model architecture and conditional denoising strategy are the main reasons for the lack of novelty contribution to TimeDiT.

**Weakness 4:** 

The authors devised a gradient derived from physical laws represented by Partial Differential Equations (PDEs) to guide the denoising process in the inference. However, the experiments related to PDEs (Table 2) demonstrate the performance of TimeDiT in solving classical partial differential equations, rather than introducing PDE theory into time series prediction. the authors do not explain in detail how this contributes to the "incorporate physics knowledge as an energy-based prior for TimeDiT during inference". The authors' claim that TimeDiT “can integrate physics-informed knowledge into time series forecasting” is not compounded, and we believe that TimeDiT's contribution to the field of time series forecasting is overstated.

**Weakness 5:** 

The experimental part generally needs improvement, there is a lack of state-of-the-art baselines to verify that TimeDiT is sufficiently competitive, and TimeDiT shows really limited performance improvement.

* Table 1 is expected to introduce the most recent baseline under each paradigm, e.g., the data-specific model TimeMixer(ICLR2024)[4], LLM-based model TimeLLM(ICLR2024), Diffusion-based model MG-TSD (ICLR2024)[3], mr-Diff(ICLR2024)[2]. In addition, an important issue is that in Table 1, Diffusion-TS(ICLR2024) exhibits far worse performance than CSDI(NIPS2019), which is a confusing result. Meanwhile, TimeDiT exhibits worse performance in PhysioNet (a) and NASDAQ compared to CSDI (NIPS2019), and TimeDiT shows less than 10% performance improvement in MIMIC-III and PhysioNet (b). Does this suggest that TimeDiT obtained from pre-training based on the masking paradigm cannot directly demonstrate competitive predictive power?

* In Table 2, the performance improvement of TimeDiT is less than 5% in most of the PDEs, e.g., the average improvement in Advection, Navier-Stokes and Diffusion Sorption is 2.27%, 2.7% and 3.88%, respectively. Further, Table 2 compares the performance of TSDiff, PPDM and PPIM for solving Partial Differential Equations only, where TSDiff as a temporal prediction model does not have a priori the ability to solve PDEs, and where PPDM and PPIM are designed for vision generation tasks. Furthermore, to the best of our knowledge, TimeDiT was not the first to propose a method for solving PDEs using diffusion models. We hope that the introduction of TimeDiT with PDE-Diffusion (ICLR2024)[5] and DiffusionPDE (ICML2024)[6] in Table 2 will help validate the effectiveness of TimeDiT.

* Similar problems still exist for the zero-shot forecasting task, and Table 3 is considered to be missing state-of-the-art baselines, such as: 1) Timer(ICLR2024), a foundation model pre-trained in multiple temporal domains, 2) GPD(2024)[7], a recent unified diffusion model for time series and TimeLLM(ICLR2024). These models also show good potential, and further experiments are considered necessary. Indeed, similar problems exist in Tables 4 and 5, where the best SOTA model shown is OneFItsALL (NIPS2023), which is significantly outdated for the rapidly evolving field.

**Reference:** 

1) Zhang, Lvmin et al. “Adding Conditional Control to Text-to-Image Diffusion Models.” 2023 IEEE/CVF International Conference on Computer Vision (ICCV) (2023): 3813-3824.

2) Shen, Lifeng et al. “Multi-Resolution Diffusion Models for Time Series Forecasting.” ICLR 2024.

3) Fan, Xinyao et al. “MG-TSD: Multi-Granularity Time Series Diffusion Models with Guided Learning Process.” ICLR 2024.

4) Wang, Shiyu et al. “TimeMixer: Decomposable Multiscale Mixing for Time Series Forecasting.” ICLR 2024.

5) Chonghan, Gao et al. "PDE-Diffusion: Physic guided diffusion model for solving partial derivative equations."

6) Huang, Jiahe et al. “DiffusionPDE: Generative PDE-Solving Under Partial Observation.” ICML 2024

7) Yang, Jiarui et al. “Generative Pre-Trained Diffusion Paradigm for Zero-Shot Time Series Forecasting.” ArXiv abs/2406.02212 (2024): n. pag.
We propose a general-purpose time series foundation model based on Diffusion Transofrmers.
@inproceedings{
anonymous2024timedit,
title={TimeDiT: General-purpose Diffusion Transformers for Time Series Foundation Model},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=FvBTy5Dz9C},
note={under review}
}
With recent advances in building foundation models for text and video data, there is a surge of interest in foundation modeling for time series.  Many families of models have been developed utilizing a temporal autoregressive Transformer architecture, whose effectiveness has been proven in Large Language Models (LLMs). However, real-world time series exhibit unique challenges, such as variable channel sizes across domains, missing values, and varying signal sampling intervals due to the multi-resolution nature of real-world data. Additionally, the unidirectional nature of temporally autoregressive decoding typically learns a deterministic mapping relationship and limits the incorporation of domain knowledge, such as physical laws. To address these challenges, we introduce the Time Diffusion Transformer (TimeDiT), a general foundation model for time series that jointly leverages the transformer inductive bias to capture temporal dependencies and the diffusion processes to generate high-quality candidate samples. The proposed mask unit for task-agnostic pretraining and task-specific sampling enables direct processing of multivariate inputs even with missing values or multi-resolution. Furthermore, we introduce a theoretically justified finetuning-free model editing strategy that allows the flexible integration of external knowledge during the sampling process. Extensive experiments conducted on a variety of tasks, such as forecasting, imputation, and anomaly detection highlight TimeDiT's adaptability as a foundation model, addressing diverse time series challenges and advancing analysis in various fields.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Time Series; Foundation model; Diffusion model
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/44517eb5f1e8d7bb476b7079d8539a5581117d17.pdf
foundation or frontier models, including LLMs
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
TimeDiT: General-purpose Diffusion Transformers for Time Series Foundation Model
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission