Structure-preserving contrastive learning for spatial time series
time series;contrastive learning;spatio-temporal data;traffic interaction
Yes
No ethics review needed.
See Weaknesses.
- The proposed method can enhance SOTA performance on various datasets.
- The writing is good, and the paper is easy to follow.
This paper proposes a time series contrastive learning loss with topology-preserving or graph-geometry preserving regularization. I appreciate the sufficient experiment made by the authors, but the novelty and originality of this paper are unclear.
- About novelty. The idea of measuring topology similarity or preserving graph geometry has been extensively studied in existing literatures. What is the key insight of this paper that is different from existing works?
- About originality. As described in Sec3.2 and Sec3.3, the proposed method adopts many existing technics, including TS2Vec loss, SoftCLT loss and topology-preserving loss. What is the origin idea or content of this work?
Yes
No ethics review needed.
1. The UEA datasets this paper used omit the two largest, InsectWingbeat and PenDigits, due to limited computation resources. Does this mean that the proposed method has limitations when dealing with large datasets?

2. See Weaknesses.
1. The topic of time series analysis is important to the ICLR community.

2. The proposed Structure-preserving contrastive learning is novel, which can enhance model performance and generalisability in downstream tasks.

3. The presentation is good and the experimental evaluation is adequate.
This paper extends time series contrastive learning by incorporating two structure-preserving regularisers: one preserves the topology of similarities between instances, and the other preserves the graph geometry of similarities across spatial and temporal dimensions. And, the proposed method preserves the structures of similarity relations more effectively and improves state-of-the-art task performances for all three tasks.
1. In terms of experimental evaluation, this paper does not analyze the efficiency of the proposed method, making the evaluation of the model incomplete.

2. In the experimental part, the ablation experiment of key modules in the model is not carried out, which makes the effectiveness of the designed module difficult to be verified.

3. In traffic prediction evaluation, some important baseline models in the field of traffic prediction were not used, making the performance comparison experiments less convincing.
Yes
No ethics review needed.
* What are the details of the model used for fine-tuning, which model is used for traffic prediction?
* For the evaluation metrics (e.g. dRMSE), how are the distances measured in datasets with spatial features vs. without spatial features? Do you account for the edge distances between connected nodes? Are they the same when calculating the structure regularizer losses (e.g. $\boldsymbol{A}$ in equation 7)?
* The graph structures in many of these datasets are static, i.e. the graph doesn’t change over time, is it still appropriate to use structure-preserving regularizers from these graph autoencoders?
* What do the colors of the nodes in Figure A1. represent? Are they the classes of the samples? Are the latent representations obtained during pre-training or after fine-tuning?
* The paper leaves many questions unanswered. The goal here is to examine the effectiveness of structure-preserving regularizers in spatiotemporal data contrastive learning, yet the paper only shows prediction improvements without explaining why one regularize works better than the other. For example, if $\mathcal{L}\_{topo}$ performs better than $\mathcal{L}\_{GGeo}$ on macroscopic traffic prediction, what does this tell us about the dataset and the loss function? Under what circumstances would $\mathcal{L}_\{GGeo}$ do better than $\mathcal{L}\_{topo}$? This could be better answered by evaluating them on the same topology dataset such as the Sphere or the Swiss Roll dataset to interpret the results better and understand the self-supervised representations. There is also no explanation on why TS2Vec performs better than SoftCLT in certain scenarios.
The question that the paper is asking is very important. Applying self-supervised representation learning techniques to multivariate time series is relatively unexplored. Many previous works focus on improving encoder architecture, and the techniques introduced in this paper are general and applicable to all deep learning models in this field.

The paper brings concepts from autoencoder regularization techniques to time series contrastive learning, forcing the distance of samples in the input space and latent space to be similar. This bridge of ideas is creative and original. The paper also introduces the background information very well, allowing the readers to fully understand the motivation behind these regularizers and how they can be used in spatiotemporal data. The overall framework is intuitive.

The paper also runs extensive experiments on two datasets, comparing the effect of using only contrastive losses and using them with the structure-preserving regularizers.
The paper aims to improve contrastive learning for spatiotemporal data. Building upon existing time series self-supervised representation learning methods, namely TS2Vec and SoftCLT, the paper adds two structure-preserving regularisers to the pre-training process. The topology-preserving loss 
$\mathcal{L}\_{topo}$
is borrowed from Topological Autoencoders by Moor et al., 2020, while the graph-geometry-preserving loss 
$\mathcal{L}\_{GGeo}$
is borrowed from Graph Geometry-Preserving Autoencoders by Lim et al., 2024.

The paper then applies six combinations of pre-training losses on the UEA datasets by Bagnall et al., 2018 and two traffic prediction datasets. In all datasets, it is evident that adding structure-preserving regularizers improves prediction performance.
* **Experimental design**: 
   * The paper lacks the "No Pre-training" baseline on the UEA datasets (this setting is included in the traffic prediction datasets). The baseline method should use only the vanilla models (i.e. models without constrastive loss or regularizers).
   * The paper does not include the training configurations (e.g. hardware specs, the GPUs/CPUs used during training, the training time, and the training/validation/test data splits ratio
    * The paper does not discuss if contrastive pre-training decreases overall training time, which is important in practice. To address this, the paper could include a plot with training time as the x-axis, test accuracy on the y-axis, and 2 curves showing 1. finetuning from scratch 2. finetuning from pre-training. For instance, see Figure 5. in Masked Autoencoders As Spatiotemporal Learners by Feichtenhofer et al., 2022.
    * Since the paper is examining the effect of contrastive learning and structure regularizers in the latent space, more baseline models should be tested in combination such as DCRNN, STGCN, Graph WaveNet, etc. These models are important spatiotemporal forecasting models that can provide more evidence for the effectiveness of the method.

* **Writing clarity**:
    * The term $r\_{\eta}$ is only explained to "regularisation against overfitting of the dynamic weights", but is not further mentioned in the methods section or hyperparameter search. 
    * It is difficult to understand what the bold and underlining mean in Tables 5, 6, 7. Perhaps adding a detailed caption for the tables helps.
    * There is no explanation on why "No-Pretraining" has the best structure preserving metrics on traffic prediction. This is counter-intuitive since the baseline does not force the distance between samples in the original and latent space to be similar.
    * In Table 1, maybe explain the abbreviations of batch size and learning rate in the caption or footnote.
    * Lines 468-472 are hard to understand without giving specific measures for improvement e.g. using GGeo + TS2Vec improved performance by x%. It is impossible to know in "These methods are also those showing sub-optimal performances in prediction", what "These methods" are specifically. The paper should provide rephrase the paragraph by giving specific improvement examples and analyzing the results in more detail. 

* **Typos**
    * Line 84-85 SOTA abbreviation should be in the introduction at the first mention
    * Line 423-424 whereas it does when together with preserving similarity structure >> whereas it does when used together with preserving similarity structure
Yes
No ethics review needed.
The novelty of this paper primarily lies in its enhancements to pre-existing methodologies. While the authors make contributions to the understanding of flow prediction, these modifications do not represent a big enough step forward in the field. Many of the techniques discussed are already prevalent in the literature, lacking the introduction of new paradigms or theoretical frameworks.
1 The paper provides a robust theoretical analysis, detailing the mechanics of flow prediction models. It includes mathematical derivations that clarify the relationships between various graph features and flow dynamics.
2 The proposed model incorporates a unique combination of simulation techniques and feature extraction from node and edge data. This method aims to closely approximate actual flow while maintaining computational efficiency.
3 The experiments are well-structured, comparing the proposed method against existing approaches. The results demonstrate the effectiveness of the model in various scenarios, showing improvements in prediction accuracy.
This article investigates structure-preserving flow prediction models, emphasizing the balance between data-driven approaches and simulation-based methods. The authors present a novel model that aims to enhance the accuracy of flow predictions by leveraging a combination of node-level and edge-level features. The proposed method is built on existing frameworks but introduces modifications to improve performance in predicting flow across graphs.
Specifically, it describes the methodology in detail, outlining how the model integrates various features such as topological, geographical, and temporal data to optimize flow predictions. It employs a two-step approach: first, it analyzes the historical flow data to identify patterns, and second, it applies simulation techniques to predict future flows based on these identified patterns. The theoretical foundation is supported by mathematical formulations that establish the relationships between flow, costs, and node attributes.
1 Despite the improvements made, the overall approach does not significantly advance the field of flow prediction. The methods employed largely build upon established techniques without introducing truly innovative concepts or frameworks.
2 The scope of the datasets used in the experiments is somewhat restricted. This limitation may impact the generalizability of the results, raising questions about how well the method would perform in diverse real-world scenarios
Preserving certain structures of similarity relations in spatio-temporal data can improve downstream task performance via contrastive learning.
@inproceedings{
anonymous2024structurepreserving,
title={Structure-preserving contrastive learning for spatial time series},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=sz7HdeVVHo},
note={under review}
}
Informative representations enhance model performance and generalisability in downstream tasks. However, learning self-supervised representations for spatially characterised time series, like traffic interactions, poses challenges as it requires maintaining fine-grained similarity relations in the latent space. In this study, we extend time series contrastive learning by incorporating two structure-preserving regularisers: one preserves the topology of similarities between instances, and the other preserves the graph geometry of similarities across spatial and temporal dimensions. We conduct experiments on multivariate time series classification, as well as macroscopic and microscopic traffic prediction. For all three tasks, our method preserves the structures of similarity relations more effectively and improves state-of-the-art task performances. This extension can be applied to an arbitrary encoder and is particularly beneficial for time series with spatial or geographical features. Our code is attached as supplementary material, which will be made openly available with all resulting data after review.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
time series contrastive learning spatio-temporal data traffic interaction
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/64447a298e40ed9090826e73f28e9877b9c06784.pdf
learning on time series and dynamical systems
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
/attachment/6ee8e1e4d81f9526bc8d3922e7c48073115aad6c.zip
Structure-preserving contrastive learning for spatial time series
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission