DROSIA: Decoupled Representation on Sequential Information Aggregation for Time Series Forecasting
decoupled representation;sequence modeling;time series forecasting;representation learning
Thank you for the comments and suggestions. We have carefully considered each point and detail our responses below:

For weaknesses:

* We would like to know more about your concerns of novelty and the technical implementation, could you please clarify that why you think our idea lacks novelty, and where of the technical implementation is weak? This would be important for us to improve our paper.

* Our motivation of this paper is to demonstrate the effectiveness and efficiency of DROSIA for sequence modeling and capturing long-distance dependencies. The reason that we choose the previously state-of-the-art channel-dependent model - iTransformer is also to demonstrate the motivation. Consequently, we believe that we should focus on various intra-channel modeling methods.

* We will upload our code, and share the link here and in the paper as soon as possible.

For questions:

* We replace the Transformer encoder of PatchTST with DROSIA encoder, which is much different when compared to Transformer, and we conduct extensive experiments and ablation studies to demonstrate the effectiveness and efficiency of DROSIA for sequence modeling and capturing long-distance dependencies.

* Because we unified the batch size and learning rate for all datasets and forecasters, and please let me explain for this.

  1. We conduct all of the experiments based on the Time-Series-Library (TSL) repo in GitHub (https://github.com/thuml/Time-Series-Library), which is open-sourced by the research team of iTransformer.

  2. TSL provides many scripts for model training. However, we found that the hyperparameters setted in the scripts of different models are not always the same, such as the number of encoder layers and attention heads, batch sizes, learning rates, and so on.

  3. We could reproduce the results of iTransformer reported in the corresponding paper using the provided scripts, however, the forecasting performance of other models (including DROSIA) could also benefit from the hyperparameters that are used for iTransformer.

  4. To conduct a fair comparison, we set all hyperparameters to be the same.

* We believe that there is no perfect model, but only applicable scenarios for the model. DROSIA is not a hexagonal warrior, but it could reach the state-of-the-art performance over all channel-independent models, and over channel-dependent models when the input lengths of time series are sufficiently long, with only linear complexity. This is enough for us to demonstrate our motivation of this paper.
Thank you for the comments and suggestions. We have carefully considered each point and detail our responses below:

For weaknesses:

* We apologize for the impression our paper has given you. Could you please specify which components were not clearly justified? This information is of great significance for us to improve the paper.

* Please see our responces to the questions.

* We will conduct a statistical test as soon as possible to demonstrate the significance of DROSIA over PatchTST in small datasets, and report the results here and in the paper for your concerns. Additionally, please allow me to reiterate the motivation of this paper.

  1. We aim to construct a simple, effective and efficient method for sequence modeling and have chosen to demonstrate its effectiveness using a variety of time series forecasting datasets.

  2. PatchTST employs the Transformer encoder for sequence modeling, and the only difference between our approach and theirs is in the encoder. However, our method outperforms PatchTST, especially on complex datasets, as evidenced by Table 1.

  3. Moreover, when longer sequences are input, DROSIA surpasses PatchTST by an even greater margin (figure 4). These results are sufficient to prove the effectiveness of our proposed method in sequence modeling and capturing long-distance dependencies, and it does so with only linear complexity.

For questions:

1. We want to balance between individual and sequential information, and reduce the computational complexity, which have been explained in the paper. This intuition is similar to the local-global architectures, and we emphasize the importance of local information. Moreover, we conduct extensive ablation studies for the comparison to potential alternatives.

2. All of the baselines are trained and evaluated with the same split settings of data, as well as the batch size, channels, input length, prediction hirizon, and so on. It is consistent with other work in the field of time series forecasting. We are confused of the "less training data" of channel-dependent models, could you please be more specific of what you mean?

3. We are very willing to introduce other important forecasters, and may have missed some models and will add them as soon as possible. However, this work focuses on the effectiveness and efficiency of DROSIA for sequence modeling and capturing long-distance dependencies, which has been demonstrated.

4. We have not evaluated the ability of DROSIA in few-shot scenarios, this is a valuable direction for us. In this paper, we focus on the effectiveness and efficiency of DROSIA for sequence modeling and capturing long-distance, rather than transfer learning.
For questions:

Firstly, we need to clarify that patch size does indeed have an impact.

* When the input length L=96 and the patch size p=32, the model’s performance declines. This is why we set the input length to 192 when analyzing the impact of various patch sizes, to enable a fairer comparison.

* We believe that the model’s predictive performance is related to the number of parameters and the depth of the network. Larger datasets require more parameters and deeper networks. When the patch size is set close to the input length, the sequence of patches is too short, leading to an insufficient number of parameters for the sequence information extractor, which is also evident from the analysis of the other three hyperparameters.

* Since the sequence information is processed by the extractor from the concatenated patch embeddings, when the dimension ratio of patch embeddings is sufficiently high, the number of extractor parameters is also sufficient. The analysis of larger dimensions could confirm this point as well, and the number of encoding layers could confirm the impact of network depth on predictive performance.

There is no parameter sharing in DROSIA.

We have tested the longer horizons in Figure 4, but not in table 5, and we are willing to do so.

Scenario "P" means only original patch embeddings are directly fed into FFN layer (equation 7), the dimension of patch embeddings is set to d for this, while d/2 for "P+S".

We are very willing to evaluate the effectiveness of DROSIA in other time series tasks, but in this paper, we focused on forecasting to demonstrate the capabilities of sequence modeling and capturing long-distance dependencies of DROSIA.

Thank you for your open-mindedness towards DROSIA. We will take your suggestions seriously and make revisions as soon as possible, hoping to gain your approval in the end.
Thank you for the comments and suggestions. We have carefully considered each point and detail our responses below:

For weaknesses:

We apologize for the inaccurate use of the term “significantly” while only reporting the average results of three trials, and will conduct a statistical test as soon as possible, report the results here and in the paper.

We believe that there is no perfect model, but only applicable scenarios for the model. We have demonstrated the effectiveness and efficiency of DROSIA in its applicable scenarios and consider that the requirements for these scenarios are not difficult to meet.

* For example, the Weather dataset includes 21 indicators, such as temperature and humidity, recorded every 10 minutes. So we can wait 1 day and 8 hours to obtain a sequence of length 192, but we need 21 different types of sensors to gather the 21 channels. Hard to imagine what express delivery could bring 21 (or even 192) types of sensors home and have them functioning properly within 1 day and 8 hours, lol. Increasing the sequence length is not necessarily more difficult than increasing the number of channels.

* Moreover, we can adjust to a finer time granularity or quickly obtain longer sequences through methods like interpolation to address this issue.

* We have honestly reported the shortcoming of DROSIA in experiments and affirm the value of inter-channel information, emphasizing the importance of efficiently modeling both intra- and inter-channel information in future work.

* DROSIA is slightly weaker than channel-dependent models only in the scenario of extremely many channels and short sequences, which might sound quite rare when described. We believe that this shortcoming of DROSIA cannot overshadow its success in other, more extensive fields.

We are very willing to position LLM-based forecasters in our study if you insist, and please allow me to explain why we haven’t done so before.

* Paper [1] (https://arxiv.org/pdf/2406.16964) thoroughly analyzed the LLM-based methods in the field of time series forecasting, and they claimed: "Popular LLM-based time series forecasters perform the same or worse than basic LLM-free ablations, yet require orders of magnitude more compute", and “Despite the recent popularity of LLMs in time series forecasting, they do not appear to meaningfully improve performance”.

* Our motivation of DROSIA is to demonstrate its effectiveness and efficient in sequence modeling and time series forecasting, so we did not compare it with LLM-based forecasters but with various transformer-based models.

* Research related to LLM requires many resources, which can be quite challenging for ordinary researchers (like me). Therefore, I hope to focus on small but meaningful work.

We will compare our model with FreTS you mentioned and other potential baselines as soon as possible, report the results here and in the paper.

We are pleased to conduct an ablation study for this component. However, the equation (7) in our paper fuse two types of information and compress the fused representation to the original dimension. It enables a full interaction between the two, and could not be simply removed in scenarios of multi-layer encoder. How would you like to conduct this ablation study?

We would upload our code, and share the link here and in the paper, add the non-averaged results of table 1 in the appendix, and fix typos, font size, and add another row “average+- standard deviation” to each table as soon as possible.

For questions:

Because of the limit of characters, we would add another official comment for responses to the questions.

References

[1] Are Language Models Actually Useful for Time Series Forecasting? Neurips, 2024
Yes
No ethics review needed.
Regarding the theory:
1. The dot product attention and the skip connection altogether are also capable of decoupling between the patch level presentation and its interaction with the whole sequence even when these two levels of presentations are summed together instead of concatenated. Can you provide a more rigorous or quantifiable definition on what the decoupling means here in the paper? 
2. Can you clarify the flow of the algorithm? For example, eq 3 through eq 7, where is S^j_1, assuming C^j is sequence level? How to get S^{j+1}_i?
3. Empirical study reveals no benefit from patching. What's the motivation behind it?

Regarding the empirical study:
1. The choice of DROSIA and other baseline methods' hparams are either unclear or arbitrary across different studies. It would be better to have more details to back a fair comparison, e.g. all methods are tuned to near optimal.
2. The lookback window for each tasks also seem a bit arbitrary, e.g. table 1 uses 96 whereas the original PatchTST paper reports 512. It would be more insightful to report multiple lookup for stronger empirical evidence.
3. Table 4 Effectiveness of DROSIA is too limited in the sense that (1) the benchmark datasets are two and high dimensional, and (2) "S" for PatchTST's original setting is a bit misleading for claiming no patch level presentation there.
4. Based on the current empirical study, it is unclear whether the performance edge in Table 1 is due to the decoupling or due to a better tuning of some linear / MLP structures which are already known to be also effective for forecasting tasks. Consider adding an appendix.
DROSIA as proposed in the paper is a practical architecture to use, and it is relatively convincing that it can deliver good performances.
In the papers the authors propose a MLP based time series forecasting framework named DROSIA which emphasizes the separate representations of patch (local) level information and sequence level information, i.e. the manual decoupling of the two. DROSIA prefers concatenation instead of summation to explicitly present the related info. They empirically benchmark DROSIA against other SotA methods and demonstrate its outstanding performance.
There are weaknesses in both the theoretical motivation and the empirical study in the paper. Novelty wise, the motivation and justification behind the decoupling of sequence and patch level information is not well supported, whereas besides this point DROSIA has no outstanding distinctions from other linear (MLP) based model. 

For the empirical study there lacks many details regarding, e.g. benchmark model parameters, reasons for setting up the benchmark parameters, etc, which makes the empirical support for the decoupling claim weak. 

Presentation wise, the writing could use some revision, specially regarding the key parts, e.g. the algorithm of DROSIA, the reasoning behind the ablation study, etc.
Yes
NA
No ethics review needed.
1. It is not clear why each component of the proposed DROSIA is designed in its current form. What advantages do the chosen design choices provide compared to potential alternatives?
2. It may not be fair to compare with channel-dependent methods e.g., iTransformer, as they have less training data.
3. Some important channel-independent and channel-dependent baselines are not mentioned or compared, such as:

[1] Unified Training of Universal Time Series Forecasting Transformers (ICML 2024)

[2] Chronos: Learning the Language of Time Series (arXiv preprint arXiv:2403.07815)

[3] One Fits All: Power General Time Series Analysis by Pretrained LM (NeurIPS 2023)

[4] S²IP-LLM: Semantic Space Informed Prompt Learning with LLM for Time Series Forecasting (ICML 2024)

[5] TimeMixer: Decomposable Multiscale Mixing for Time Series Forecasting (ICLR 2024)

4. What is the performance of DROSIA in few-shot scenarios compared to the baselines?
* The paper is well-written and organized.
* The proposed network architecture appears to be novel.
* The method is lightweight, utilizing relatively fewer parameters than existing approaches.
This paper presents Decoupled Representation On Sequential Information Aggregation (DROSIA) for time series forecasting. The key idea is to aggregate sequential information in a decoupled fashion, effectively balancing it with individual point information. The experimental results demonstrate the effectiveness of the proposed method.
* There are clarity issues in explaining the proposed method. Some components and their roles are not clearly justified.
* Certain comparisons may not be entirely fair. In addition, several related works are not mentioned or compared (see the questions below).
* For datasets with a small number of variates or low complexity, the improvements over other models, such as PatchTST, are marginal.
Yes
Nan
No ethics review needed.
- What is the big difference between your work and PatchTST? 
- You consider iTransformer as a strong baseline, but the results in Table 1 seem to be different from the results in iTransformer. Any experimental settings changed?
- Efficiency analysis in Table 3 should include more linear-based methods: DLinear, and TiDE. It would be great to conduct more analysis like Figure 10 in iTransformer rather than only computational complexity.
- Nice paper writing.
- Enough ablation study is also included.
For the task of time series forecasting, authors propose decoupled representation to integrate temporal relationships once as an additional representation for each point, achieving sequential information aggregation in a decoupled fashion.
- The idea of decoupled representation lacks novelty. The technical implementation is relatively weak.
- Lack of in-depth analysis, e.g., channel-dependent model.
- No available codes.
Yes
No ethics review needed.
Do you have any intuition for certain behaviours in your sensitivity analysis? e.g., why patch size is not important, why for larger dataset in terms of number of channels e.g., Traffic high values of almost all hyper-parameters makes things worse? 

Is there any parameter sharing in DROSIA? 

Are observations made in table 5 going to hold for longer horizons e.g., H=720?  

What is “P” model in table 4 last column? 

Compared baselines are also applicable to other time-series tasks including classification, short-term forecasting, imputation, and anomaly detection and often competitive in said tasks, but DROSIA is only studied in long-term forecasting do you have any sense on applicability of DROSIA in other time series tasks?

I would be happy to revise my score if authors clarify questions/weaknesses particularly:
- Missing experiments e.g., ablation or potentially missing baselines as well as information to evaluate performance better e.g., standard deviation or significance test. 
- Issue with number of features/channels and input length
- Authors have done extensive experiments ranging from benchmarks to complexity
- DROSIA achieved better performance with relatively simpler model in Long-term forecasting task
Authors proposed DROSIA a method of time-series forecasting that incorporates both point-wise and temporal information by applying the following steps:
- Patching similar to existing methods e.g., PatchTST
- DROSIA encoding 
    - Sequence aggregation I.e., applying multiple encoding layers on vectorized patches 
    - Information extraction I.e., a MLP on concatenated representation of the previous step 
    - Representation fusion i.e., layer normalization of concatenated representations of information extraction + vectorized patches followed by a fully connected transformation (similar to residual connections) 
- Decoding (projection) i.e., making predictions 

Authors studied performance of DROSIA on state-of-the-art of time series forecasting benchmarks and compared with some of the well-known methods in this area.
Although average over 3 trials is reported, standard deviations are not reported. Reporting standard deviation is crucial when performance gap is small. Particularly, when authors claim “significantly outperforming” a method this needs to be confirmed by conducting a statistical test e.g., t-test or Wilcoxon test (based on assumptions).  

The proposed method needs adjustment in input length to outperform iTransformer particularly on datasets with a lot of variables and shorter horizons e.g., 96 which could be a disadvantage in applicability of the proposed method on real-world applications. Authors did not provide any instruction on how to find “sufficiently long” input length for their method. 

Recently LLM-based methods for time series forecasting have shown state-of-the-art performance [1-4] some of which also based on patching [2] but there is no indication of this category in neither related works nor compared methods. Examples are: 
Just to be clear, I am not asking authors to compare with all of these LLM-based methods but I’d like to know their at least their thoughts on positioning this line of work in their study. 

Authors have compared their method with DLinear which is based on MLP that utilizes point-wise information but there are also MLP-based models such as [5] that incorporate global and local information which to me are more similar to the proposed method but is missing from compared methods. 

One missing ablation experiment is related to the equation (7). What is the performance without this component? 

Some reproducibility information is missing such as code (cited in the abstract that it will be provided later), learning rate or any utilized regularizations. 

Original (non-averaged) results of table 1 should be provided in the appendix 


Smaller fixes: 

- Typo on page 4 “In Equation (4)” -> Equation (3)

- Typo on page 5 “In Equation (3) -> Equation (8)

- Font size in figures is too small (at least for me)

- It would be helpful to add another row “average+- standard deviation” to each table to summarize the results per analysis

References 

[1] Large language models are zero-shot time series forecasters. Neurips, 2023.

[2] One Fits All: Power General Time Series Analysis by Pretrained LM. Neurips, 2023.

[3] Time-LLM: Time Series Forecasting by Reprogramming Large Language Models. ICLR, 2024.

[4] TEST: Text Prototype Aligned Embedding To Activate LLM’S Ability for Time Series. ICLR, 2024.

[5] Frequency-domain MLPs are More Effective Learners in Time Series Forecasting. Neurips, 2023.
We propose DROSIA to integrate temporal relationships as additional representation for each time point, which achieves the sequential information aggregation in a decoupled manner.
@inproceedings{
anonymous2024drosia,
title={{DROSIA}: Decoupled Representation on Sequential Information Aggregation for Time Series Forecasting},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=6SxOzYVuy6},
note={under review}
}
Time series forecasting is crucial in various fields, including finance, energy consumption, weather, transportation, and network traffic. It necessitates effective and efficient sequence modeling to encapsulate intricate temporal relationships. However, conventional methods often aggregate sequential information into representations of each time point by considering other points in the sequence, thereby ignoring the intra-individual information and suffering from inefficiency. To address these challenges, we introduce a novel approach, DROSIA: Decoupled Representation On Sequential Information Aggregation, which only integrates temporal relationships once as an additional representation for each point, achieving sequential information aggregation in a decoupled fashion. Thus balancing between individual and sequential information, along with a reduction in computational complexity. We select several widely used time series forecasting datasets, and previously top-performing models and baselines, for a comprehensive comparison. The experimental results validate the effectiveness and efficiency of DROSIA, which achieves state-of-the-art performance with only linear complexity. When provided with sufficiently long input data, the channel-independent DROSIA even outperforms the current best channel-dependent model, highlighting its proficiency in sequence modeling and capturing long-distance dependencies. Our code will be made open-source in the subsequent version of this paper.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
decoupled representation sequence modeling time series forecasting representation learning
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/b51f2a8a53cea5c2fc1a2e622ebcc72193713832.pdf
unsupervised, self-supervised, semi-supervised, and supervised representation learning
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
DROSIA: Decoupled Representation on Sequential Information Aggregation for Time Series Forecasting
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission