Forecasting chaotic systems with zero-shot learning
chaos;nonlinear dynamics;forecasting;physics;scientific machine learning
**[Initial clarifying questions at start of discussion period]**

Thank you for your review! We are currently putting together our experiments based on your comments. We had two clarifying questions that would help us design our experiments:

+ Q2: My understanding is that Chronos does not use positional encodings, but rather tokenizes time series and then defers embedding to an LLM. Since the trained Chronos models use T5 (encoder-decoder), they use relative positional embedding rather than rotary/sinusoidal. Since relative positional embeddings are learned during training, we can’t ablate them without retraining Chronos. Besides describing this in greater detail in the text, are there other experiments  the referee would like to see here?

+ Q4: I agree that this is a highly-informative point of comparison, but so far we haven’t gotten strong results from our attempts to fine-tune Chronos. When we fine-tune Chronos with the original hyperparameters, the loss plateaus very quickly. My best guess is that we need to go up several orders of magnitude in compute and dataset size to see improvements, which would approach the scale of retraining Chronos. I expect that new experiments that meaningfully improve Chronos would require a level of effort and resources at the scale of training a new foundation model. In our revision, we can describe these limitations and our initial efforts in greater detail in the text; would the referee find this sufficient?

**We therefore plan during the discussion period to add several new experiments addressing Q1 and Q3, along with clarifying language regarding Q2 and Q4. Would the referee find a revision along these lines sufficient?**

Thank you again for reviewing our paper.
Clarifying questions at start of discussion period
Yes
NA
No ethics review needed.
1. Impact of Initial Conditions: Can the authors elaborate on how initial conditions affect Chronos’s zero-shot forecasting stability? Is there a significant dependency on starting points, especially when far from the attractor’s typical state?
2. Effectiveness of Positional Encodings: How critical is the choice of positional encodings for Chronos when applied to chaotic systems? Have other encoding methods been tested, and if so, how did they affect both short-term accuracy and long-term attractor preservation?
3. Model Complexity and Forecasting Horizon: Given that larger models perform better in long-term forecasting, could the authors provide insights into the mechanisms by which increased model size aids in managing chaotic variability?
4. Comparison with In-Weights Fine-Tuning: Since the study focuses on zero-shot learning, have the authors considered comparing this approach with in-weights fine-tuning on a few chaotic trajectories to assess the performance improvement?
1. Extensive Benchmarking: The authors conduct a thorough and extensive evaluation, with a dataset that includes 135 chaotic systems, providing statistically robust results.
2. Foundation Model Scalability: By demonstrating the scalability of Chronos across varying chaotic systems, the study establishes the potential of foundation models for scientific machine learning in challenging domains.
3. Zero-Shot Learning Feasibility: The findings validate that foundation models can make meaningful forecasts in chaotic systems with minimal domain-specific adjustments, a promising step for generalizing AI across diverse scientific domains.
4. Insight into Long-Term Statistical Consistency: The study’s focus on long-term attractor behavior offers a novel way to assess model performance, moving beyond conventional short-term forecast accuracy.
The article explores the application of foundation models, particularly zero-shot learning, to forecast chaotic systems. Using the foundation model Chronos, the study investigates whether these models can generate accurate short-term forecasts and capture the long-term statistical properties of chaotic attractors, which are characteristic of chaotic systems. Through an extensive evaluation on 135 chaotic systems, the authors found that, even without specific training on chaotic dynamics, Chronos provides competitive forecasts, preserving geometric and statistical attributes of chaotic attractors over the long term. The study presents these findings as a benchmark for the feasibility and limitations of using foundation models in scientific machine learning, especially in physics-informed tasks.
1. The article mentions that Chronos’s forecast accuracy is sensitive to initial conditions, but the specifics of this dependency aren’t deeply explored. Could the authors provide a more systematic investigation into how initial condition variability impacts model robustness? For example, have you considered evaluating Chronos’s performance across a range of initial conditions sampled from different regions of the attractor, particularly comparing accuracy in central versus peripheral areas? Additionally, could you quantify how forecast accuracy shifts as the initial conditions deviate from those in the training data? Such analyses would offer valuable insights into Chronos’s generalization capabilities and help identify any specific sensitivities related to initial condition variability.
2. Positional Encoding and Temporal Structure: Given the chaotic nature of the systems studied, the choice of positional encodings (e.g., rotary embeddings) could significantly impact model performance, especially in maintaining temporal coherence over long horizons. However, the article lacks an in-depth discussion or ablation study regarding the choice of these encodings.
3. Limited Explanation of Performance in Extreme Cases: The authors mention that larger models tend to perform better, yet details on why model size improves stability in chaotic systems are sparse. A theoretical or empirical justification of this scaling behavior, such as through model capacity for capturing non-linear dynamics, would improve understanding.
4. Scalability of Model Parameters and Computational Requirements: Although Chronos shows promise in forecasting chaotic systems, the article could discuss practical limitations regarding computational costs, especially for larger model sizes, and the trade-offs compared to specialized models like NBEATS or TiDE.
Yes
No ethics review needed.
The authors aim to evaluate the proposed approach on real-world data? Interesting characteristics can be observed. 

How does foundations models forecast accuracy vary with different initial conditions within chaotic systems? This sensibility limits its generalization? 

I suggest the authors to state limitations of the zero-shot long-term prediction problem. Another possible limitation is how the foundation model zero-shot approach will behave when there is a drastic different dynamical behaviour between training and testing.
The paper presents strong findings on the power of foundation models for zero-shot learning in chaotic systems. Firstly, it demonstrates that foundation models can perform as well as fully trained models across multiple scenarios, especially in long-term forecasting.

Another strength of the paper is its study and evaluation of the suitability of foundation models in the field of scientific machine learning. This proof of concept can definitely lead to new developments in the study of chaotic systems.
The paper presents a large-scale evaluation of the zero-shot learning capabilities of foundation models for chaotic systems. The authors investigate both short-term and long-term forecasting scenarios. They evaluate the foundation models against over 100 chaotic systems and observe several interesting findings. The key point is that foundation models not only generalize to new initial conditions but also to entirely new systems. This suggests the models have the capacity to capture the broader dynamical structures of chaotic systems.
Further experiments are needed to assess sensitivity to initial conditions and real-world chaotic systems, as these factors currently weaken the proposed evaluation and findings.

While the application and findings of the paper are important, I believe the work in its current state lacks novelty.
Yes
No ethics review needed.
-
- The core contribution is a benchmark of foundation models on the evolution of different chaotic systems. I think this is a good contribution which goes into a similar direction as Chronos and ForecastPFN which use synthetic data (Chronos not exclusively) to pretrain their models and still obtain good performance. I agree with a reference in the conclusion stating that there appears to be common language shared by time-series as it's surprising that the existing synthetic data generators would result in such well performing models.
- I see potential for this benchmark to be used itself for more expressive synthetic data in time-series foundation models.
- I think the finding that chaotic systems can be learned better by foundation models from context rather than by a specific model is valuable and interesting.
The present paper proposes to use time-series generated from chaotic systems as a new benchmark in time-series forecasting.
It makes the case for time-series foundation models (Chronos) which were pretrained on diverse synthetic and real-world data which it finds to yield better results than customly trained algorithms. This is surprising given that Chronos is a univariate prediction model and therefore sees the x, y, z components of the chaotic systems independently.
- It would be good to give more details on the LSTM and transformer models shown in the paper in order to allow for a fair comparison. I couldn't find details on the number of layers used e.g. for LSTM after hyperparameter optimization like the number of layers or the state size of the LSTM.
- Could you clarify your rational on choosing the models you evaluated in the paper? Given the recent interest in linear RNNs like Mamba 1 / Mamba 2 / Gated Linear Attention, could you provide some results in this model class? Maybe Mamba 2, as it's fast to train.
- I think the paper in its current state is quite verbose and could benefit from a clearer structure. For example sections 5.1 and 5.2 could have individual paragraphs with titles indicating the finding which the authors would like to emphasize.
Large language models zero-shot forecast chaotic dynamics
@inproceedings{
anonymous2024forecasting,
title={Forecasting chaotic systems with zero-shot learning},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=TqYjhJrp9m},
note={under review}
}
Time-series forecasting is a challenging task that traditionally requires specialized models custom-trained for the specific task at hand. Recently, inspired by the success of large language models, foundation models pre-trained on vast amounts of time-series data from diverse domains have emerged as a promising candidate for general-purpose time-series forecasting. The defining characteristic of these foundation models is their ability to perform zero-shot learning, that is, forecasting a new system from limited context data without explicit re-training or fine-tuning. Here, we evaluate whether the zero-shot learning paradigm extends to the challenging task of forecasting chaotic systems. Across $135$ distinct chaotic dynamical systems and $10^8$ timepoints, we find that foundation models produce competitive forecasts compared to custom-trained models (including NBEATS, TiDE, etc.), particularly when training data is limited. Interestingly, even after point forecasts fail, foundation models preserve the geometric and statistical properties of the chaotic attractors, demonstrating a surprisingly strong ability to capture the long-term behavior of chaotic dynamical systems. Our results highlight the promises and pitfalls of foundation models in making zero-shot forecasts of chaotic systems.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
chaos nonlinear dynamics forecasting physics scientific machine learning
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/da30c46de0c771752a9ab101288628bc4d0b51ef.pdf
applications to physical sciences (physics, chemistry, biology, etc.)
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
/attachment/f40ae21027b3c04fa77ce03f7db942c5e6efe9ab.zip
Forecasting chaotic systems with zero-shot learning
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission