Time Series Representation Models for Multivariate Time Series Forecasting and Imputation
time series;attention;forecasting;imputation;multivariate
Yes
No ethics review needed.
Listed above in the weaknesses section.
- The paper focuses on an important problem of time series forecasting and imputation which can be useful in several practical use cases.
- Experiments show that the framework is able to outperform baselines on multiple datasets on the time series forecasting task.
- The paper is easy to follow.
The paper introduces a novel architecture called Time Series Representation Model (TSRM) for multivariate time series forecasting and imputation. The TSRM architecture is based on a Transformer encoder-like configuration with hierarchically ordered encoding layers.  The TSRM architecture integrates representation learning within a multilayered model, where each layer features a distinct representation learning module. Each representation learning layer independently captures representations at a different hierarchical level, restoring the original input dimensions to enable hierarchical stacking of layers independent of the input dimension. The proposed approach achiever similar or better performance than state-of-the-art approaches on forecasting and imputation tasks while significantly reducing complexity in the form of learnable parameters.
- One major concern is the motivation behind certain architectural choices, such as the introduction of merge layers and the inclusion of representation layers in each hierarchical level. The ablation studies in Figure 2 show that removing the merge layers results in only a minor drop in performance  (0.379 -> 0.382 and 0.161 -> 0.165) compared to other ablation experiments, which raises questions about their necessity. Additionally, it is unclear whether having multiple representation layers is more effective than using a single set of CNN layers at the beginning of the Transformer architecture, as is common in computer vision. Without clear evidence of performance improvements, the novelty of the proposed approach appears marginal.
- Furthermore, the claim that each representation learning layer captures representations at a different hierarchical level requires qualitative analysis to support it. 
- The paper's claim of achieving state-of-the-art performance on imputation tasks is not supported by the results in Table 2.
- How are masks handled in CNN layers (representation block)?
- Does the proposed approach need to be trained separately for different prediction horizons and different missing ratios or is it generalizable?
Yes
No ethics review needed.
Generally, the technique of the manuscript is partially sound, and some major concerns are listed below:
How is the kernel size chosen—is it based on specific criteria, such as the period length, or selected empirically? What about the sensitivity of this parameter? The results using $K$ independent 1D CNN layers with varying kernel sizes have not been compared to those with a uniform kernel size, even in the ablation study, which seems more important than discussing $N$.
1. The article emphasizes model interpretability which is crucial for understanding and trusting its predictions.
2.  The article includes links to the source code, enabling researchers to reproduce the results.
3. The article demonstrates the model's effectiveness and generalization capabilities by testing it on datasets from various domains.
The paper introduces a multilayered representation learning architecture called TSRM designed for multivariate time series forecasting and imputation. TSRM utilizes hierarchical encoding layers, each with a representation layer to capture diverse temporal patterns and an aggregation layer for combining learned representations. Based on a Transformer encoder-like configuration with self-attention mechanisms, TSRM outperforms SOTA approaches on seven benchmark datasets.
Generally, the technique of the manuscript is partially sound, and some major concerns are listed below:

1. The title of this paper could be clearer. It does not convey the specific techniques or strengths used, nor does it highlight the unique aspects of the work. The authors claim that this method can better handle different time scales, so perhaps emphasizing this could be beneficial. And this is just a suggestion.
2. The paper emphasizes that the targeted tasks are forecasting and imputation. However, the representation learning method in this paper does not specifically address forecasting and imputation, which may require additional clarification. Alternatively, future work could explore extending the method to other tasks.
3. There are too many subsections in the related work, leading to some redundant content. In particular, the "Few/zero-shot learning" subsection covers methods unrelated to this paper, so it could be appropriately condensed.
Yes
No ethics review needed.
Please see weaknesses.
1) What is $\alpha$ in Eq 4?
2) How does TSRM generalize to different missing ratios at test time?
* Clear presentation and informative figure 1.
* Experiments on both forecasting/imputation. 
* TSRM are lighter than popular models such as TimesNet/Patch-TST in terms of parameter-counts.
This paper introduces new architecture for forecasting/imputation of time-series. Overall, it introduces encoder blocks comprising of a Representation layer and Merge layer with self-attention in between. The representation and merge layers have no non-linearities and consist of convolution/transposed-convolution layers respectively. The models are evaluated on forecasting/imputation tasks.
* The paper introduces a lot of hyperparameters. Based on Appendix A.4, the paper searches over 13 possible EncLayers $\times$ 5 possible attention-heads $\times$ 5 possible feature-embedding sizes $\times$ 2 possible attention-functions $\times$ (at least) 4 possible CNN configurations = 2600 configurations! 
* From the table having best hyperparameters, sparse-attention seems to be the best combination across all datasets. But, there seems to be significant variation across different datasets/horizons. Even for the same dataset, different horizon seems to have a different optimal configuration. 
* Although TSRM is lighter than other models and this may help accelerate the search, the computational cost of picking the right hyperparameters cannot be ignored. Also, I couldn't find any analysis of wallclock times (train/inference) in the paper. 
* Table 7 reports the Mean/Std across 5 runs of ETTh2 and weather. From my understanding, this mean/std is computed for running the model with optimal configuration for 5 times. In practice, the hyperparameters should be selected based on 5 runs and then, the corresponding test-performance should be reported. However, this may be very expensive given the hyperparameter space. 
* TimesNet seems to be uniformly better than TSRM on the imputation task. Also, PatchTST is not considered for the imputation task. 
* The model is trained on forecasting/imputation tasks only. Self-supervised learning may also be explored to establish the generality of this architecture. However, the hyperparameter-search may make this expensive. 
* Given the huge hyperparameter space of TSRM, it may be fairer to allow PatchTST to select its patch-size/input-length. Under optimal Patch-TST performance, does TSRM still perform favorably?
Yes
No ethics review needed.
Please check limitations.
The methodology presentation is clear, the code is provided, and the performance surpasses multiple models.
This paper proposes a multilayered representation learning architecture for multivariate time series forecasting and imputation, achieving superior performance on seven public datasets.
1. **Unclear motivation:** The paper does not discuss which limitations of existing methods it aims to address, lacking a direct explanation in this regard.  
2. **Limited scope of tasks:** Although the paper proposes a representation learning approach, it is only applied to forecasting and imputation, which are tasks of the same nature. I hope to see applications in classification and anomaly detection as well.  
3. **Lack of discussion and discussion on most related works:** The paper compares only with end-to-end models, but it misses comparisons with the most relevant representation-based methods, such as Cost[1]. Additionally, recent foundation models, such as MOMENT[3] and Moirai[2], could also be leveraged as representations. Please discuss differences and make comparisions.

[1] Woo, Gerald, et al. "CoST: Contrastive Learning of Disentangled Seasonal-Trend Representations for Time Series Forecasting." International Conference on Learning Representations.


[2] Woo, Gerald, et al. "Unified Training of Universal Time Series Forecasting Transformers." Forty-first International Conference on Machine Learning.


[3] Goswami, Mononito, et al. "MOMENT: A Family of Open Time-series Foundation Models." Forty-first International Conference on Machine Learning.
A multilayered representation leraning architecture for multivariate time series forecasting and imputation.
@inproceedings{
anonymous2024time,
title={Time Series Representation Models for Multivariate Time Series Forecasting and Imputation},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=UCeZMMyjm2},
note={under review}
}
We introduce a multilayered representation learning architecture called Time Series Representation Model (TSRM) for multivariate time series forecasting and imputation. The architecture is structured around hierarchically ordered encoding layers, each dedicated to an independent representation learning task. Each encoding layer contains a representation layer designed to capture diverse temporal patterns and an aggregation layer responsible for combining the learned representations. The architecture is fundamentally based on a Transformer encoder-like configuration, with self-attention mechanisms at its core. The TSRM architecture outperforms state-of-the-art approaches on seven established benchmark datasets for both forecasting and imputation tasks while significantly reducing complexity in the form of learnable parameters. The source code is available at https://anonymous.4open.science/r/TSRM-D7BE.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
time series attention forecasting imputation multivariate
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/52c4a47c23265fda2282ffb576e17a519dc05c33.pdf
unsupervised, self-supervised, semi-supervised, and supervised representation learning
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
Time Series Representation Models for Multivariate Time Series Forecasting and Imputation
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission