Enhancing Multi-Modal Reasoning Over Time-Series and Natural Language Data
Time-Series;Natural Language Processing;Multi-Modal
Yes
No ethics review needed.
- For Table 3: Why TS training could improve the BBH and GPQA results while harming the MMLU-Pro results? Is it just performance fluctuation or there is an inner difference between these benchmarks?
- Since the dataset is generated by visual QA, I’m wondering if multi-modal (vision + text) pretraining & SFT could also handle the downstream benchmark evaluations.
- Open question: what affects the time-series data pretraining? e.g., context length, the vision ability, the reasoning ability, etc.
- Typo:
    - Line 049: LLava → LLaVA
    - Line 057: it’s effectiveness → its effectiveness
    - Line 327: LLama3.1 → LLaMA 3.1
- This paper constructs a novel time-series+text dataset for multi-modal instruction tuning. The construction method is interesting and there includes four kinds of conversations.
- The time-series pretraining does not affect text-based benchmark results and may improve the reasoning (BBH).
This paper combines time-series data with text to train a chatbot with numerical data understanding abilities. This enables directly interact with raw data without modality conversion. To deal with the task, the authors propose to apply vision-language models to generate synthesized instruction tuning datasets. Experiments show time-series data pretraining does not affect the original text-based evaluations.
Please feel free to discuss and leave comments.

- There lacks of dataset quality check. Since the instruction tuning dataset is generated by GPT-4o-mini with images, it may introduce errors. Although the authors have filtered out bad cases (as introduced in the appendix), it is not sure whether hallucinations do not exist. Maybe I missed some details, please feel free to have a discussion.
- Limited modality interleaving: since the trained model could only generated text (as the dataset constructed to do so), it limits the application to time-series data understanding, and ignores the time-series forecasting. Maybe a more fine-grained instruction dataset should be constructed with multi-modal responses.
- Lack of baselines: Maybe visual language models could be considered as baselines to take images and instructions as input and output their options. This could be a reference to tell readers why numerical data pretraining is useful for relevant time-series data understanding.
Yes
No ethics review needed.
None
The paper identifies a relatively underexplored area in the field of time-series analysis by focusing on the integration of natural language processing with time-series data, suggesting that further research is needed to fully leverage the potential of multi-modal reasoning in this context.
This paper introduces Chat-TS, a large language model (LLM) designed for multi-modal reasoning over time-series and natural language data. It addresses the gap in existing time-series models' ability to integrate textual information for decision-making. The authors contribute three new datasets for training and evaluating LLMs on time-series reasoning tasks: the TS Instruct Training Dataset, the TS Instruct QA Benchmark, and the TS Instruct Qualitative Benchmark. They also propose a training strategy that enhances the LLM's reasoning capabilities without compromising its natural language proficiency. Evaluation results demonstrate Chat-TS's state-of-the-art performance in multi-modal reasoning while maintaining strong language skills. Although the paper showcases promising results, the experimental section could benefit from more rigorous analysis and a deeper exploration of the model's limitations.
It appears that the manuscript has some formatting issues, particularly with the margins not conforming to ICLR's template specifications, and it could benefit from a more in-depth analysis of the model's limitations, potential biases, and practical applications, as well as from more rigorous experimental validation against current methodologies.
Yes
No ethics review needed.
1. Why does the base model show strong performance on the TS Instruct QA benchmark but perform less well on TS Math Analysis and Decision Making tasks?

2. Since you describe this work as the development of a new model family, do you expect that the improvements from your method would still hold with larger models? A scaling experiment with a larger model would be beneficial, if feasible.

3. Typo in Line 23: "qualitative" should be "quantitative."

4. I found it difficult to follow the process for using GPT-4 to create the dataset. Does this process is prompting GPT4o for generating responses based on a tailored system prompt, along with the image of the time series, metadata such as length, and number of channels? If so, is this essentially a form of distillation using GPT-4 to produce the dataset?
1. The discrete tokenizer proposed in this paper offers a straightforward yet effective approach to transforming continuous time-series data into a form compatible with LLMs. This innovative approach could potentially inspire future work on bridging time-series analysis with language models, expanding the versatility of LLMs in temporal domains.

2. The authors present a comprehensive synthetic datasets including benchmarks specifically tailored for time-series tasks, which includes three new resources: 1) TS Instruct Training Dataset for instruction tuning on time-series data, 2) TS Instruct QA Benchmark for evaluating multimodal reasoning in time-series contexts, and 3) TS Instruct Quantitative Benchmark for a detailed assessment across QA, mathematical reasoning, and decision-making questions.

3. Performance on Text-Only Tasks: Despite the focus on time-series tasks, the model also maintains strong performance on traditional text-only tasks, such as those in the MMLU benchmark. This versatility indicates that the proposed methods do not compromise the model's capabilities in established LLM domains, making the approach both flexible and robust.
This paper introduces a method for leveraging LLM in time-series reasoning and question-answering (QA) tasks. The authors propose a discrete tokenizer that transforms continuous time-series data into sequences of discrete bins, facilitating better integration of time-series data with LLMs. The paper also introduces three new datasets: (1) a time-series instruction tuning dataset; (2) a time-series QA dataset designed to assess multimodal reasoning capabilities in time-series tasks; and (3) a TS-Instruct Quantitative benchmark for evaluating performance in QA, mathematical reasoning, and decision-making beyond accuracy alone. The authors' best-performing model demonstrates strong results relative to a baseline model, though some experiment outcomes are mixed.
- Generalization: It is unclear whether the synthetic dataset can generalize effectively across diverse types of time-series data or to sequences longer than those encountered during training. Experiments regarding this concern is highly recommended to be added.

- Dataset Alignment Process: The alignment between time-series data and natural language relies solely on GPT-4, without any additional quality control to validate this alignment process. This raises concerns about potential inconsistencies or inaccuracies in the dataset.

- Soundness and Baselines:
    - Reporting results from common multimodal baseline models would provide readers with a clearer understanding of the current state-of-the-art performance for these tasks. This is particularly relevant, as the base model shows very strong performance in the detailed quantitative evaluation presented in Table 5.
    - Given that GPT-4o is heavily involved in the dataset creation, a direct performance comparison with GPT-4o is missing.
    - Mixed Experimental Results: The experimental results are somewhat mixed (as detailed in Question), which makes it difficult to conclusively assess the model's strengths and limitations.
Yes
No ethics review needed.
-
1. The paper provides a detailed quantitative analysis that assesses the quality of model responses, going beyond simple accuracy metrics.

2. The proposed discrete tokenizer for multivariate time-series data requires no training while maintaining a very high reconstruction quality (in terms of MSE on the validation set).

3. The proposed Chat-TS models show an average improvement in reasoning capabilities of ~15% on the new TS Instruct QA benchmark compared to existing methods.
This paper introduces a novel approach to leverage multi-modal large language models (LLMs) for time-series analysis. It focuses on maintaining the LLM's reasoning capabilities and utilizing its extensive knowledge base to enhance time-series data reasoning. Specifically, the authors develop Chat-TS, a new foundation model capable of explicitly preserving the general knowledge and reasoning abilities of LLMs, as well as a training strategy that integrates time-series tokens into the LLM vocabulary, enhancing the model's reasoning ability over both text and time-series modalities without compromising its core natural language capabilities. Besides, the authors curate a new time-series instruction tuning dataset (TS Instruct) and a new testbed for benchmarking multi-modal reasoning in time-series analysis (TS Instruct QA).
While the TS Instruct dataset and the TS Instruct QA benchmark are comprehensive, they primarily consist of synthetic data. Real-world time-series data can be more complex and noisy, which might affect the model's performance in practical applications.
@inproceedings{
anonymous2024enhancing,
title={Enhancing Multi-Modal Reasoning Over Time-Series and Natural Language Data},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=Pik26bc4Jx},
note={under review}
}
Time-series analysis is critical in many industries such as healthcare, finance and energy sectors, where understanding time-series trends alongside contextual information is essential for informed decision making. However, current time-series models are limited in their ability to perform reasoning that involves both time-series data and textual information. In this work we address this gap by introducing Chat-TS, a large language model (LLM)  designed specifically for reasoning over time-series and textual data. Unlike traditional time-series models Chat-TS integrates time-series tokens into the LLM vocabulary, enhancing its reasoning ability over both text and time-series modalities without compromising its core natural language capabilities.
To support the development and validation of Chat-TS we contribute three new datasets: the TS Instruct Training Dataset which pairs diverse time-series data with relevant text instructions and responses for instruction tuning, the TS Instruct question and answer (QA) benchmark, a set of nearly 4000 multiple-choice questions designed to evaluate multi-modal reasoning and the TS Instruct Qualitative Benchmark which provides a smaller subset of QA, math and decision making questions for LLM evaluation. Our training strategy preserves the inherent reasoning capabilities of the LLM while augmenting it with time-series reasoning capabilities. Evaluation results show that Chat-TS achieves state-of-the-art performance in multi-modal reasoning tasks, maintaining strong natural language proficiency while advancing time-series reasoning. All models, datasets, and code will be made publicly available [link].
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Time-Series Natural Language Processing Multi-Modal
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/6d272325f60e17bc55085acb9792c54d8acbd9e3.pdf
foundation or frontier models, including LLMs
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
/attachment/a5f0d1d60e1be7e16ed5b774da799dd435058b17.zip
Enhancing Multi-Modal Reasoning Over Time-Series and Natural Language Data
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission