Evaluating Large Language Models on Time Series Feature Understanding: A Comprehensive Taxonomy and Benchmark
Elizabeth Fons Rachneet Kaur Soham Palande Zhen Zeng Svitlana Vyetrenko Tucker Balch

Simulating Ultrafast Transient Absorption Spectra from First Principles using a Time-Dependent Configuration Interaction Probe
    Benjamin G. Levine
    April 29, 2024
==============================================================================================================================






Large Language Models (LLMs) offer the potential for automatic time series analysis and reporting, which is a critical task across many domains, spanning healthcare, finance, climate, energy, and many more. In this paper, we propose a framework for rigorously evaluating the capabilities of LLMs on time series understanding, encompassing both univariate and multivariate forms. We introduce a comprehensive taxonomy of time series features, a critical framework that delineates various characteristics inherent in time series data. Leveraging this taxonomy, we have systematically designed and synthesized a diverse dataset of time series, embodying the different outlined features. This dataset acts as a solid foundation for assessing the proficiency of LLMs in comprehending time series. 
Our experiments shed light on the strengths and limitations of state-of-the-art LLMs in time series understanding, revealing which features these models readily comprehend effectively and where they falter. In addition, we uncover the sensitivity of LLMs to factors including the formatting of the data, the position of points queried within a series and the overall time series length.




§ INTRODUCTION



Time series analysis and reporting play a crucial role in many areas like healthcare, finance, climate, etc. With the recent advances in Large Language Models (LLMs), integrating them in time series analysis and reporting processes presents a huge potential for automation. Recent works have adapted general-purpose LLMs for time series understanding in various specific domains, such as seizure localization in EEG time series <cit.>, cardiovascular disease diagnosis in ECG time series <cit.>, weather and climate data understanding <cit.>, and explainable financial time series forecasting <cit.>.

Despite these advancements in domain-specific LLMs for time series understanding, it is crucial to conduct a systematic evaluation of general-purpose LLMs' inherent capabilities in generic time series understanding, without domain-specific fine-tuning. This paper aims to uncover the pre-existing strengths and weaknesses in general-purpose LLMs regarding time series understanding, such that practitioners can be well informed of areas where the general-purpose LLMs are readily applicable, and focus on areas for improvements with targeted efforts during fine-tuning.

To systematically evaluate the performance of general-purpose LLMs on generic time series understanding, we propose a taxonomy of time series features for both univariate and multivariate time series. This taxonomy provides a structured categorization of core characteristics of time series across domains. Building upon this taxonomy, we have synthesized a diverse dataset of time series covering different features in the taxonomy. This dataset is pivotal to our evaluation framework, as it provides a robust basis for assessing LLMs' ability to interpret and analyze time series data accurately. Specifically, we examine the state-of-the-art LLMs' performance across a range of tasks on our dataset, including time series features detection and classification, data retrieval as well as arithmetic reasoning. 

Our contributions are three-fold:

    
  * Taxonomy - we introduce a taxonomy that provides a systematic categorization of important time series features, an essential tool for standardizing the evaluation of LLMs in time series understanding.
    
  * Diverse Time Series Dataset - we synthesize a comprehensive time series dataset, ensuring a broad representation of the various types of time series, encompassing the spectrum of features identified in our taxonomy.
    
  * Evaluations of LLMs - our evaluations provide insights into what LLMs do well when it comes to understanding time series and where they struggle, including how they deal with the format of the data, where the query data points are located in the series and how long the time series is.




§ RELATED WORK




 §.§ Large Language Models

Large Language Models (LLMs) are characterized as pre-trained, Transformer-based models endowed with an immense number of parameters, spanning from tens to hundreds of billions, and crafted through the extensive training on vast text datasets <cit.>. Notable examples of LLMs include Llama2 <cit.>, PaLM <cit.>, GPT-3 <cit.>, GPT4 <cit.>, and Vicuna-13B <cit.>. 
These models have surpassed expectations in numerous language-related tasks and extended their utility to areas beyond traditional natural language processing. For instance, <cit.> have leveraged LLMs for the prediction and modeling of human mobility, <cit.> for explainable financial time series forecasting, and <cit.> for seizure localization. This expansive application of LLMs across diverse domains sets the stage for their potential utility in the analysis of time series data, a domain traditionally governed by statistical and machine learning models.



 §.§ Language models for time series

Recent progress in time series forecasting has capitalized on the versatile and comprehensive abilities of LLMs, merging their language expertise with time series data analysis. This collaboration marks a significant methodological change, underscoring the capacity of LLMs to revolutionize conventional predictive methods with their advanced information processing skills. In the realm of survey literature, comprehensive overviews provided by <cit.> and <cit.> offer valuable insights into the integration of LLMs in time series analysis, highlighting key methodologies, challenges, and future directions. Notably, <cit.> have set benchmarks for pre-trained LLMs such as GPT-3 and Llama2 by assessing their capabilities for zero-shot forecasting. Similarly, <cit.> introduced Prompcast, and it adopts a novel approach by treating forecasting as a question-answering activity, utilizing strategic prompts. Further, <cit.> delved into the potential of LLMs for generating explainable forecasts in financial time series, tackling inherent issues like cross-sequence reasoning, integration of multi-modal data, and interpretation of results, which pose challenges in conventional methodologies. Additionally, <cit.> demonstrated that leveraging frozen pre-trained language models, initially trained on vast corpora, for time series analysis that could achieve comparable or even state-of-the-art performance across various principal tasks in time series analysis including imputation, classification and forecasting. 



 §.§ LLMs for arithmetic tasks

Despite their advanced capabilities, LLMs face challenges with basic arithmetic tasks, crucial for time series analysis involving quantitative data <cit.>. 
Research has identified challenges such as inconsistent tokenization and token frequency as major barriers <cit.>. Innovative solutions, such as Llama2's approach to digit tokenization <cit.>, highlight ongoing efforts to refine LLMs' arithmetic abilities, enhancing their applicability in time series analysis. 



§ TIME SERIES DATA




 §.§ Taxonomy of Time Series Features



Our study introduces a comprehensive taxonomy for evaluating the analytical capabilities of Large Language Models (LLMs) in the context of time series data. This taxonomy categorizes the intrinsic characteristics of time series, providing a structured basis for assessing the proficiency of LLMs in identifying and extracting these features. Furthermore, we design a series of datasets following the proposed taxonomy and we outline an evaluation framework, incorporating specific metrics to quantify model performance accurately across various tasks.













The proposed taxonomy encompasses critical aspects of time series data that are frequently analyzed for different applications. Table <ref> shows the selected features in increasing complexity, and each sub-feature. We evaluate the LLM in this taxonomy in a two-step process. In first place, we evaluate if the LLM can detect the feature, and in a second step, we evaluate if the LLM can identify the sub-category of the feature. A detailed description of the process is described in Sec. <ref>.



 §.§ Synthetic Time Series Dataset



Leveraging our taxonomy, we construct a diverse synthetic dataset of time series, covering the features outlined in the previous section. 
We generated in total 9 datasets with 200 time series samples each. Within each dataset the time series length is randomly chosen between 30 and 150 to encompass a variety of both short and long time series data. In order to make the time series more realistic, we add a time index, using predominantly daily frequency. Fig. <ref> showcases examples of our generated univariate time series.
Each univariate dataset showcases a unique single-dimensional patterns, whereas multivariate data explore series interrelations to reveal underlying patterns. Please see Table <ref> in the appendix for examples of each univariate dataset, and Table <ref>  for visual examples of the multivariate cases. 
For a detailed description of the generation of each dataset, refer to Sec. <ref> in the Appendix. 






§ TIME SERIES BENCHMARK TASKS


Our evaluation framework is designed to assess the LLMs' capabilities in analyzing time series across the dimensions in our taxonomy (Sec. <ref>).
The evaluation includes four primary tasks:



  
Feature Detection
This task evaluates the LLMs' ability to identify the presence of specific features within a time series, such as trend, seasonality, or anomalies. For instance, given a time series dataset with an upward trend, the LLM is queried to determine if a trend exists. 
Queries are structured as yes/no questions to assess the LLMs' ability to recognize the presence of specific time series features, such as "Is a trend present in the time series?"



  
Feature Classification
Once a feature is detected, this task assesses the LLMs' ability to classify the feature accurately. For example, if a trend is present, the LLM must determine whether it is upward, downward, or non-linear.
This task involves a QA setup where LLMs are provided with definitions of sub-features within the prompt. Performance is evaluated based on the correct identification of sub-features, using the F1 score to balance precision and recall. This task evaluates the models' depth of understanding and ability to distinguish between similar but distinct phenomena.



  
Information Retrieval Evaluates the LLMs' accuracy in retrieving specific data points, such as values on a given date.



  
Arithmetic Reasoning Focuses on quantitative analysis tasks, such as identifying minimum or maximum values. Accuracy and Mean Absolute Percentage Error (MAPE) are used to measure performance, with MAPE offering a precise evaluation of the LLMs' numerical accuracy.

Additionally, to account for nuanced aspects of time series analysis, we propose in Sec. <ref> to study the influence of multiple factors, including time series formatting, location of query data point in the time series and time series length.



§ PERFORMANCE METRICS AND FACTORS





 §.§ Performance Metrics

We employ the following metrics to report the performance of LLMs on various tasks.


  
F1 Score Applied to feature detection and classification, reflecting the balance between precision and recall.


  
Accuracy Used for assessing the information retrieval and arithmetic reasoning tasks.


  
Mean Absolute Percentage Error (MAPE) Employed for numerical responses in the information retrieval and arithmetic reasoning tasks, providing a measure of precision in quantitative analysis.



 §.§ Performance Factors


We identified various factors that could affect the performance of LLMs on time series understanding, for each we designed deep-dive experiments to reveal the impacts.



  
Time Series Formatting 

Extracting useful information from raw sequential data as in the case of numerical time series is a challenging task for LLMs. The tokenization directly influences how the patterns are encoded within tokenized sequences <cit.>, and methods such as BPE separate a single number into tokens that are not aligned.
On the contrary, Llama2 has a consistent tokenization of numbers, where it splits each digit into an individual token, which ensures consistent tokenization of numbers <cit.>. 
We study different time series formatting approaches to determine if they influence the LLMs performance to capture the time series information. In total we propose 9 formats, ranging from simple CSV to enriched formats with additional information. 




  
Time Series Length

We study the impact that the length of the time series has in the retrieval task. Transformer-based models use attention mechanisms to weigh the importance of different parts of the input sequence. Longer sequences can dilute the attention mechanism's effectiveness, potentially making it harder for the model to focus on the most relevant parts of the text <cit.>.



  
Position Bias
Given a retrieval question, the position of where the queried data point occurs in the time series might impact the retrieval accuracy. Studies have discovered recency bias <cit.> in the task of few-shot classification, where the LLM tends to repeat the label at the end. Thus, it's important to investigate whether LLM exhibits similar bias on positions in the task of time series understanding.



§ EXPERIMENTS




 §.§ Experimental setup




  §.§.§ Models

We evaluate the following LLMs on our proposed framework: 1) GPT4. <cit.> 2) GPT3.5. 3) Llama2-13B <cit.>, and 4) Vicuna-13B <cit.>. We selected two open-source models, Llama2 and Vicuna, each with 13 billion parameters, the version of Vicuna is 1.5 was trained by fine-tuning Llama2. Additionally we selected GPT4 and GPT3.5 where the number of parameters is unknown. In the execution of our experiments, we used an Amazon Web Services (AWS) g5.12xlarge instance, equipped with four NVIDIA A10G Tensor Core GPUs, each featuring 24 GB of GPU RAM. This setup was essential for handling both extensive datasets and the computational demands of LLMs.




  §.§.§ Prompts



The design of prompts for interacting with LLMs is separated into two approaches: retrieval/arithmetic reasoning and detection/classification questioning.



  
Time series characteristics
To evaluate the LLM reasoning over time series features, we use a two-step prompt with an adaptive approach, dynamically tailoring the interaction based on the LLM's responses. The first step involves detection, where the model is queried to identify relevant features within the data. If the LLM successfully detects a feature, we proceed with a follow-up prompt, designed to classify the identified feature between multiple sub-categories. 
For this purpose, we enrich the prompts with definitions of each sub-feature (e.g. up or down trend), ensuring a clearer understanding and more accurate identification process. 
An example of this two-turn prompt is shown in Fig. <ref>. The full list can be found in Sec. <ref> of the supplementary.



  
Information Retrieval/Arithmetic Reasoning
We test the LLM's comprehension of numerical data represented as text by querying it for information retrieval and numerical reasoning, as exemplified in Fig. <ref> and detailed in the supplementary Sec. <ref>.







 §.§ Benchmark Results


In Table <ref>, we display the main results for all tasks outlined in Sec. <ref>. 



The results for univariate time series feature detection and classification tasks illustrate GPT4's robustness in trend and seasonality detection, substantially outperforming Llama2 and Vicuna. However, the detection of structural breaks and volatility presents challenges across all models, with lower accuracy scores.

GPT4 excels in trend classification tasks, demonstrating superior performance. However, in classifying seasonality, outliers, and structural breaks, performance is mixed, with Vicuna sometimes surpassing Llama2, highlighting the distinct strengths of each model. Figure <ref> summarizes the accuracy performance for the information retrieval and arithmetic reasoning tasks, and F1 score for the feature detection and classification tasks for all models. 

In multivariate time series feature detection and classification tasks, all models achieve moderate accuracy, suggesting potential for enhancement in intricate multivariate data analysis.

For information retrieval tasks, GPT4 outperforms GPT3.5 and other models, achieving perfect accuracy in identifying the value on a given date. It also maintains a low Mean Absolute Percentage Error (MAPE), indicative of its precise value predictions. The arithmetic reasoning results echo these findings, with GPT4 displaying superior accuracy, especially in determining minimum and maximum values within a series.







 §.§ Deep Dive on Performance Factors




  
Time Series Formatting

We present four formatting approaches in this section, , which is a common comma separated value,  where the time series is formatted as  for each pair date-value. We also use the formatting approach proposed by <cit.> which we denominate  that adds blank spaces between each digit of the time series, tokenizing each digit individually, and , an enriched format where we add a column to the time series with arrows indicating if the value has moved up, down or remained unchanged. Examples of every approach can be found in Sec. <ref> in the Appendix. 

Table <ref> shows the results for the four time series formatting strategies. For the full results, please refer to Tables <ref> and <ref>. For the information retrieval and arithmetic reasoning tasks, the  formatting yields better results across all models. This approach provides more structure to the input, and outperforms other formats in a task where the connection between time and value is important. For the detection and classification tasks, the  formatting does not yield better results. Interestingly the  formatting that adds an additional column to the time series yields better results in the trend classification task. This means that the LLMs can correctly map the symbol to the time series movement and use it to achieve the best performance in trend classification. Furthermore, GPT3.5 leverages this additional information in the trend and anomalies datasets but not in the seasonality dataset.









  
Time Series Length

Figure <ref> shows the performance of GPT3.5, Llama2 and Vicuna on three datasets, ,  and  wich have time series with different lengths. We observe that GPT3.5 retrieval performance degrades slowly with increasing sequence length. Llama2 and and Vicuna suffer a more steep degradation especially from time series of length 30 steps to 60 steps; for longer sequences the degradation in performance becomes linear. 






  
Position Bias
We carry out a series of experiments to determine how the  position of the target value affects task performance across various types of time series data. We address progressively more complex objectives: 1) identifying the presence of a value in a time series without a specified date (<ref>); 2) retrieving a value corresponding to a specific date (<ref>); and 3) identifying the minimum and maximum values (<ref>). 
We cover a range of time series data, from monotonic series without noise to those with noise, sinusoidal patterns, data featuring outliers (spikes), and Brownian motion scenarios, each adding a layer of complexity. We examine how the position of the target value within the four quadrants — 1st, 2nd, 3rd, and 4th— affects the efficacy of these tasks across the varied time series landscapes. This approach helps reveal the influence of position on different LLMs (GPT3.5, Llama2, and Vicuna) in the task of time series understanding.

We consider the presence of position bias when the maximum performance gap between quadrants exceeds 10%. 
Given this criterion, our analysis provides the following key takeaways on position bias impacting LLM performance across the defined tasks:

    
  * Pronounced position bias is observed across all tasks and LLMs
    
        
  * GPT models show significant bias exclusively in complex tasks that involve arithmetic reasoning.
        
  * Both Llama2 and Vicuna demonstrate position biases across all tasks, from the simplest to the most complex ones.
    
    
  * The degree of complexity in the time series data tends to increase the extent of position bias observed within each task.

Refer to Section <ref> in the appendix, where we offer a detailed analysis of position bias across each task to further substantiate these conclusions.



§ CONCLUSION

In conclusion, we provide a critical examination of general-purpose Large Language Models (LLMs) in the context of time series understanding. Through the development of a comprehensive taxonomy of time series features and the synthesis of a diverse dataset that encapsulates these features, we have laid a solid foundation for evaluating the capabilities of LLMs in understanding and interpreting time series data. Our systematic evaluation sheds light on the inherent strengths and limitations of these models, offering valuable insights for practitioners aiming to leverage LLMs in time series understanding. Recognizing the areas of weakness and strength in general-purpose LLMs' current capabilities allows for targeted enhancements, ensuring that these powerful models can be more effectively adapted to specific domains.


§ LIMITATIONS

In this section, we detail the key limitations of our study and suggest pathways for future research.

Time series data frequently intersects with data from other domains. In the financial industry, for instance, analysis often combines time series data like stock prices and transaction volumes with supplementary data types such as news articles (text), economic indicators (tabular), and market sentiment analysis (textual and possibly visual). Our future work aims to delve into how LLMs can facilitate the integration of multimodal data, ensure cohesive data modality alignment within the embedding space, and accurately interpret the combined data insights.

Currently, our application of LLMs in time series analysis is primarily focused on comprehending time series features. However, the lack of interpretability mechanisms within our framework stands out as a significant shortcoming. Moving forward, we plan to focus on developing and integrating interpretability methodologies for LLMs specifically tailored to time series data analysis contexts.



§ ACKNOWLEDGEMENTS


This paper was prepared for informational purposes by the Artificial Intelligence Research group of JPMorgan Chase & Co and its affiliates (“J.P. Morgan”) and is not a product of the Research Department of J.P. Morgan.  J.P. Morgan makes no representation and warranty whatsoever and disclaims all liability, for the completeness, accuracy or reliability of the information contained herein.  This document is not intended as investment research or investment advice, or a recommendation, offer or solicitation for the purchase or sale of any security, financial instrument, financial product or service, or to be used in any way for evaluating the merits of participating in any transaction, and shall not constitute a solicitation under any jurisdiction or to any person, if such solicitation under such jurisdiction or to such person would be unlawful.  












§ SYNTHETIC TIME SERIES DATASET




 §.§ Univariate Time Series

The primary characteristics considered in our univariate dataset include:


  * Trend We generated time series data to analyze the impact of trends on financial market behavior. This dataset encompasses linear and quadratic trends. For linear trends, each series follows a simple linear equation a * t + b, where a (the slope) varies between 0.1 and 1, multiplied by the direction of the trend, and b (the intercept) is randomly chosen between 100 and 110. This simulates scenarios of steadily increasing or decreasing trends. For quadratic trends, the series is defined by a * t^2 + b * t + c, with a varying between 0.01 and 0.05 (again adjusted for trend direction), b between 0 and 1, and c between 0 and 10, or adjusted to ensure non-negative values. The quadratic trend allows us to simulate scenarios where trends accelerate over time, either upwards or downwards, depending on the direction of the trend. This approach enables the exploration of different types of trend behaviors in financial time series, from gradual to more dynamic changes, providing a comprehensive view of trend impacts in market data.


  * Seasonality In our study, we meticulously crafted a synthetic dataset to explore and analyze the dynamics of various types of seasonality within time series data, aiming to closely mimic the complexity found in real-world scenarios. This dataset is designed to include four distinct types of seasonal patterns, offering a broad spectrum for analysis: (1) Fixed Seasonal Patterns, showcasing regular and predictable occurrences at set intervals such as daily, weekly, or monthly, providing a baseline for traditional seasonality; (2) Varying Amplitude, where the strength or magnitude of the seasonal effect fluctuates over time, reflecting phenomena where seasonal influence intensifies or diminishes; (3) Shifting Seasonal Pattern, characterized by the drift of seasonal peaks and troughs over the timeline, simulating scenarios where the timing of seasonal effects evolves; and (4) Multiple Seasonal Patterns, which presents a combination of different seasonal cycles within the same series, such as overlapping daily and weekly patterns, to capture the complexity of real-world data where multiple seasonalities interact. This diverse dataset serves as a foundation for testing the sensitivity and adaptability of analytical models to detect and quantify seasonality under varying and challenging conditions.


  * Anomalies and outliers refer to observations that significantly deviate from the typical pattern or trend observed in the dataset. The types of outliers included in our generated dataset are: 1) single sudden spike for isolated sharp increases, 2) double and triple sudden spikes for sequences of consecutive anomalies, 3) step spike and level shift for persistent changes, and 4) temporal disruption for sudden interruptions in the pattern. We also include a no outlier category as a control for comparative analysis. Parameters such as the location and magnitude of spikes, the duration and start of step spikes, the placement and size of level shifts, and the initiation and conclusion of temporal disruptions are randomly assigned to enhance the dataset's diversity and relevance.


  * Structural breaks
in time series data signify substantial changes in the model generating the data, leading to shifts in parameters like mean, variance, or correlation. These are broadly classified into two types: parameter shifts and regime shifts, with a third category for series without breaks. 
Parameter shifts involve changes in specific parameters such as mean or variance, including sub-types like mean shifts, variance shifts, combined mean-variance shifts, seasonality amplitude shifts, and autocorrelation shifts. 
Regime shifts represent deeper changes that affect the model's structure, including: distribution changes (e.g., normal to exponential), stationarity changes (stationary to non-stationary), linearity changes (linear to non-linear models), frequency changes, noise trend changes, error correlation changes, and variance type changes. 
The occurrence of these shifts is randomly determined within the time series.


  * Volatility 
We generated synthetic time series data to simulate various volatility patterns, specifically targeting clustered volatility, leverage effects, constant volatility, and increasing volatility, to mimic characteristics observed in financial markets. For clustered volatility, we utilized a GARCH(1,1) model with parameters ω=0.1, α=0.05, and β=0.9, ensuring the sum of α and β remained just below 1 for stationarity, thus capturing high volatility persistence. To simulate the leverage effect, our model increased volatility in response to negative returns, reflecting typical market dynamics. Additionally, we created time series with constant volatility by adding normally distributed random noise (standard deviation of 1) to a cumulative sum of random values, and with increasing volatility by scaling the noise in proportion to the increasing range of the series (scaling factor up to 5 towards the end of the series). The latter was achieved by multiplying the standard deviation of the random noise by a linearly increasing factor, resulting in a volatility profile that progressively intensified. These methodologies enabled us to comprehensively represent different volatility behaviors in financial time series, including constant, increasing, clustered, and leverage-induced volatilities, thereby enriching our analysis with diverse market conditions.


  * Statistical properties
Next, we constructed a dataset to delve into significant features of time series data, centering on fat tails and stationarity. The dataset sorts series into four categories: those exhibiting fat tails, characterized by a higher likelihood of extreme values than in a normal distribution; non-fat-tailed, where extreme values are less probable; stationary, with unchanging mean, variance, and autocorrelation; and non-stationary series. Non-stationary series are further divided based on: 1) changing mean: series with a mean that evolves over time, typically due to underlying trends.
2) changing variance: series where the variance, or data spread, alters over time, suggesting data volatility.
3) seasonality: series with consistent, cyclical patterns occurring at set intervals, like seasonal effects.
4) trend and seasonality: series blending both trend dynamics and seasonal fluctuations.




 §.§ Multivariate Time Series

For our analysis, we confined each multivariate series sample to include just 2 time series. The main features of our generated multivariate dataset encompass:


  * Correlation involves analyzing the linear relationships between series, which is crucial for forecasting one time series from another when a correlation exists. The randomly selected correlation coefficient quantifies the strength and direction of relationships as positive (direct relationship), negative (inverse relationship), or neutral (no linear relationship) between series.


  * Cross-correlation evaluates the relationship between two time series while considering various time lags, making it valuable for pinpointing leading or lagging relationships between series. For our data generation, the time lag and correlation coefficient are randomly chosen.


  * Dynamic conditional correlation  focuses on scenarios where correlations between series vary over time.  The points in the time series at which correlation shifts take place are selected randomly.




 §.§ Data Examples

c


Trend 

4cm

    < g r a p h i c s >

(a) Linear trend

4cm

    < g r a p h i c s >

(b) Quadratic trend

4cm

    < g r a p h i c s >

(c) Exponential trend

4cm

    < g r a p h i c s >

(d) No clear trend


 


Seasonality 

4cm

    < g r a p h i c s >

(a) Fixed seasonality

4cm

    < g r a p h i c s >

(b) Varying amplitude 

4cm

    < g r a p h i c s >

(c) Shifting patterns 

4cm

    < g r a p h i c s >

(d) Multiple Seasonalities


 


Volatility 

4cm

    < g r a p h i c s >

(a) Constant volatility 

4cm

    < g r a p h i c s >

(b) Increasing volatility 

4cm

    < g r a p h i c s >

(c) Clustered volatility

4cm

    < g r a p h i c s >

(d) No volatility 


 


Anomalies and Outliers 

4cm

    < g r a p h i c s >

(a) Double sudden spikes

4cm

    < g r a p h i c s >

(b) Step spike

4cm

    < g r a p h i c s >

(c) Level shift 

4cm

    < g r a p h i c s >

(d) Temporal Disruption


 


Structural breaks 

4cm

    < g r a p h i c s >

(a) Parameter shift 
 (change in variance)

4cm

    < g r a p h i c s >

(b) Parameter shift 
 (change in seasonality amplitude) 

4cm

    < g r a p h i c s >

(c) Regime shift 
 (noise trend change)

4cm

    < g r a p h i c s >

(d) Regime shift 
 (stationarity change)


 


Statistical properties 
 
4cm

    < g r a p h i c s >

(a) Fat tailed

4cm

    < g r a p h i c s >

(b) Non-stationary 
 (trend)

4cm

    < g r a p h i c s >

(c) Non-stationary 
 (changing variance over time)

4cm

    < g r a p h i c s >

(d) Non-stationary 
 (seasonality)



Examples of the generated univariate time series. The x- and y-axis are intentionally omitted to focus exclusively on the shape and characteristics of the time series.





c

 Correlation
 

5cm

    < g r a p h i c s >

(a) Positive correlation

5cm

    < g r a p h i c s >

(b) Negative correlation

5cm

    < g r a p h i c s >

(c) No correlation





 Cross-correlation 
 

5cm

    < g r a p h i c s >



(a) Lagged positive correlation

5cm

    < g r a p h i c s >



(b) Lagged negative correlation





Dynamic conditional correlation 

 
5cm

    < g r a p h i c s >



(a) Positive correlation 
 (first half)

5cm

    < g r a p h i c s >



(b) Negative correlation 
 (first half)

5cm

    < g r a p h i c s >



(d) Negative correlation 
 (second half)



Examples of the generated multivariate
time series. The x- and y-axis are intentionally omitted to focus exclusively on the shape and characteristics of the time series.







§ ADDITIONAL DATASETS


Brownian Data: We generate a synthetic time series dataset exhibiting brownian motion. The data consists of 400 samples where each time series has a length of 175. We control for the quadrant in the which the maximum and minimum values appear using rejection sampling i.e. there are 50 samples for which the maximum value in the time series occurs in the first quadrant, 50 samples for which the maximum value appears in the second quadrant, and so on, upto the fourth quadrant. In a similar manner we control for presence of the minimum value in each quadrant. 
Outlier Data: We generate a synthetic time series dataset where each time series contains a single outlier which is the either the minimum or maximum values in the time series. The data consists of 400 samples where each time series has a length of 175. We control for the quadrant in the which the maximum and minimum (outlier) values appear using rejection sampling i.e. there are 50 samples for which the maximum value in the time series occurs in the first quadrant, 50 samples for which the maximum value appears in the second quadrant, and so on, upto the fourth quadrant. In a similar manner we control for presence of the minimum value in each quadrant. 
Monotone Data: We generate a synthetic time series dataset where each time series is monotonically increasing or decreasing. The data consists of 400 samples (200 each for increasing/decreasing) where each time series has a length of 175.  
Monotone (with Noise) Data: We generate a synthetic time series dataset where each time series is increasing or decreasing. The data consists of 400 samples (200 each for increasing/decreasing) where each time series has a length of 175. Note that dataset is different from the Monotone data as the time series samples are not strictly increasing/decreasing.



§ ADDITIONAL RESULTS




 §.§ Trend










 §.§ Seasonality









 §.§ Anomalies











 §.§ Volatility








Position Bias 

 Does the position of the target value affect the performance of identifying its presence in various types of time series data? 
Refer to Figure <ref>, which includes a confusion matrix (with `1: yes' indicating presence of the number in the series and `0: no' indicating its absence) and bar plot showing the accuracy in each quadrant for each LLM and type of time series data.

GPT achieves nearly perfect performance across all quadrants and time series types, indicating an absence of position bias in detecting the presence of a number within the time series. 
Llama2 does not exhibit position bias in monotonic series without noise but begins to show position bias as the complexity of the time series increases, such as in monotonic series with noise and sinusoidal series. We believe this bias is also present in Brownian series; however, due to the higher complexity of the dataset, Llama2's performance is poor across all quadrants, making the impact of the bias less discernible.
Vicuna displays superior performance compared to Llama2 across all datasets but continues to exhibit position bias. Notably, this bias appears in most datasets, such as monotonic series without noise, sinusoidal series, and Brownian motion series.

c


GPT 3.5 

4cm

    < g r a p h i c s >


4cm

    < g r a p h i c s >


4cm

    < g r a p h i c s >


4cm

    < g r a p h i c s >



 


4cm

    < g r a p h i c s >



(a) Monotonic (no noise)

4cm

    < g r a p h i c s >

(b) Monotonic with noise

4cm

    < g r a p h i c s >

(c) Sinusoidal

4cm

    < g r a p h i c s >

(d) Brownian motion


 


Llama2 

4cm

    < g r a p h i c s >


4cm

    < g r a p h i c s >


4cm

    < g r a p h i c s >


4cm

    < g r a p h i c s >



 
4cm

    < g r a p h i c s >

(a) Monotonic (no noise)

4cm

    < g r a p h i c s >

(b) Monotonic with noise

4cm

    < g r a p h i c s >

(c) Sinusoidal

4cm

    < g r a p h i c s >

(d) Brownian motion


 


Vicuna 

4cm

    < g r a p h i c s >


4cm

    < g r a p h i c s >


4cm

    < g r a p h i c s >


4cm

    < g r a p h i c s >



 
4cm

    < g r a p h i c s >

(a) Monotonic (no noise)

4cm

    < g r a p h i c s >

(b) Monotonic with noise

4cm

    < g r a p h i c s >

(c) Sinusoidal

4cm

    < g r a p h i c s >

(d) Brownian motion



Confusion matrix and accuracy by quadrant for the search task


    



Does the position impact the retrieval performance for a specific date's value from time series data?
Refer to Figure <ref> for bar plots that illustrate the accuracy across each quadrant. 

Once again, GPT achieves nearly perfect performance across all quadrants and time series types, suggesting no position bias in the retrieval task either. Similar to the findings in <ref>, Vicuna outperforms Llama2. Moreover, both Vicuna and Llama2 exhibit position bias in most datasets, including monotonic series both with and without noise, and sinusoidal series.

c


GPT 3.5 

4cm

    < g r a p h i c s >



(a) Monotonic (no noise)

4cm

    < g r a p h i c s >

(b) Monotonic with noise

4cm

    < g r a p h i c s >

(c) Spikes 

4cm

    < g r a p h i c s >

(d) Brownian motion


 


Llama2 

4cm

    < g r a p h i c s >

(a) Monotonic (no noise)

4cm

    < g r a p h i c s >

(b) Monotonic with noise

4cm

    < g r a p h i c s >

(c) Spikes 

4cm

    < g r a p h i c s >

(d) Brownian motion





Vicuna 

4cm

    < g r a p h i c s >

(a) Monotonic (no noise)

4cm

    < g r a p h i c s >

(b) Monotonic with noise

4cm

    < g r a p h i c s >

(c) Spikes 

4cm

    < g r a p h i c s >

(d) Brownian motion


 
Confusion matrix and accuracy by quadrant for the retrieval task






Does the position impact the efficiency of identifying minimum and maximum values in different types of time series data?
Refer to Figure <ref> for bar charts illustrating the accuracy distribution across quadrants. 

For the first time, GPT models show position bias in the spikes dataset, attributed to the increased complexity of the task, which involves arithmetic reasoning. Llama2 exhibits position bias in most datasets, notably in monotonic series with noise, spikes, and Brownian motion series. Vicuna also demonstrates position bias in most datasets, including monotonic series both with and without noise, as well as spikes series.

c


GPT 3.5 

4cm

    < g r a p h i c s >



(a) Monotonic (no noise)

4cm

    < g r a p h i c s >

(b) Monotonic with noise

4cm

    < g r a p h i c s >

(c) Spikes 

4cm

    < g r a p h i c s >

(d) Brownian motion


 


Llama2 

4cm

    < g r a p h i c s >

(a) Monotonic (no noise)

4cm

    < g r a p h i c s >

(b) Monotonic with noise

4cm

    < g r a p h i c s >

(c) Spikes 

4cm

    < g r a p h i c s >

(d) Brownian motion





Vicuna 

4cm

    < g r a p h i c s >

(a) Monotonic (no noise)

4cm

    < g r a p h i c s >

(b) Monotonic with noise

4cm

    < g r a p h i c s >

(c) Spikes 

4cm

    < g r a p h i c s >

(d) Brownian motion


 
Confusion matrix and accuracy by quadrant for the min-max extraction task. Note that monotonic series can have maximum or minimum values only in the first or fourth quadrant.











§ TIME SERIES FORMATTING




Custom

[basicstyle=,columns=fullflexible]
"Date|Value2020-01-01|1002020-01-02|1052020-01-03|1032020-01-04|103"


[basicstyle=,columns=fullflexible]
Date|Value
2020-01-01|100
2020-01-02|105
2020-01-03|103
2020-01-04|103


TSV

[basicstyle=,columns=fullflexible]
"Date2020-01-011002020-01-021052020-01-031032020-01-04103"



[basicstyle=,columns=fullflexible,keepspaces=true]
Date        Value
2020-01-01  100
2020-01-02  105
2020-01-03  103
2020-01-04  103



Plain

[basicstyle=, breaklines=true, tabsize=2, breakatwhitespace=false]
"Date: 2020-01-01, Value: 100: 2020-01-02, Value: 105: 2020-01-03, Value: 103: 2020-01-04, Value: 103"



[basicstyle=,columns=fullflexible,keepspaces=true]
Date: 2020-01-01, Value: 100
Date: 2020-01-02, Value: 105
Date: 2020-01-03, Value: 103
Date: 2020-01-04, Value: 103


JSON

[basicstyle=, breaklines=true, tabsize=2, breakatwhitespace=false]
"Date":"2020-01-01","Value":100"Date":"2020-01-02","Value":105"Date":"2020-01-03","Value":103"Date":"2020-01-04","Value":103


[basicstyle=,columns=fullflexible,keepspaces=true]
"Date":"2020-01-01","Value":100
"Date":"2020-01-02","Value":105
"Date":"2020-01-03","Value":103
"Date":"2020-01-04","Value":103



Markdown

[basicstyle=, breaklines=true, tabsize=2, breakatwhitespace=false]
"|Date|Value||—|—||2020-01-01|100||2020-01-02|105||2020-01-03|103||2020-01-04|103|"



[basicstyle=,columns=fullflexible,keepspaces=true]
|Date|Value|
|—|—|
|2020-01-01|100|
|2020-01-02|105|
|2020-01-03|103|
|2020-01-04|103|


Spaces

[basicstyle=, breaklines=true, tabsize=2, breakatwhitespace=false]
"Date,Value2020-01-01,1 0 02020-01-02,1 0 52020-01-03,1 0 32020-01-04,1 0 3"



[basicstyle=,columns=fullflexible,keepspaces=true]
Date,Value
2020-01-01,1 0 0
2020-01-02,1 0 5
2020-01-03,1 0 3
2020-01-04,1 0 3



Context

[basicstyle=, breaklines=true, tabsize=2, breakatwhitespace=false]
"Date,Value2020-01-01,[100]2020-01-02,[105]2020-01-03,[103]2020-01-04,[103]"



[basicstyle=,columns=fullflexible,keepspaces=true]
Date,Value
2020-01-01,[100]
2020-01-02,[105]
2020-01-03,[103]
2020-01-04,[103]


Symbol

[basicstyle=, breaklines=true, tabsize=2, breakatwhitespace=false, escapeinside=(*@@*)]
"Date,Value,DirectionIndicator2020-01-01,100,(*@@*)2020-01-02,105,(*@@*)2020-01-03,103,(*@@*)2020-01-04,103,(*@@*)"


[basicstyle=, breaklines=true, escapeinside=(*@@*)]
Date,Value,DirectionIndicator
2020-01-01,100,(*@@*)
2020-01-02,105,(*@@*)
2020-01-03,103,(*@@*)
2020-01-04,103,(*@@*)



Base/csv

[basicstyle=, breaklines=true, tabsize=2, breakatwhitespace=false]
"Date,Value2020-01-01,1002020-01-02,1052020-01-03,1032020-01-04,103"



[basicstyle=,columns=fullflexible,keepspaces=true]
Date,Value
2020-01-01,100
2020-01-02,105
2020-01-03,103
2020-01-04,103




 §.§ Additional results of time series formatting

















§ PROMPTS




Seasonality Prompts


Prompt 1: Detection

"Input:.

Question: can you detect any cyclic or periodic patterns in this time series? Only answer 'Yes' or 'No'."

Prompt 2: Classification

"Given the following definitions:

Fixed seasonal patterns: Regular, predictable patterns occurring at fixed intervals (e.g., daily, weekly, monthly).

Varying amplitude: Seasonal patterns where the magnitude of the seasonal effect changes over time.

Shifting seasonal pattern: When the timing of the seasonal pattern shifts over time.

Multiple seasonal pattern: Presence of more than one seasonal pattern, such as daily and weekly patterns.

Select one of the following answers:

(a) the time series has a fixed seasonal pattern, (b) the time series has seasonal pattern with varying amplitude, (c) the time series has a shifting seasonal pattern, (d) the time series has multiple seasonal patterns.

Only answer (a), (b), (c) or (d)"




Anomaly Prompts

"Input:.

Prompt 1: Detection

Question: can you detect any irregularities in this time series? Only answer 'Yes' or 'No'."

Prompt 2: Classification

"Given the following definitions:

Outlier: data point that notably deviates from the overall pattern of the data.

Step-spike: sudden, sustained change in the data level, followed by a return to the original baseline.

Level shift: sudden and lasting change in the average value of the series.Temporal disruption:  interval where data is missing or not recorded.

Select one of the following answers that best describes the provided time series:

(a) the time series has one or more outliers, (b) the time series has a step-spike, (c) the time series has a level shift, (d) the time series has a temporal disruption.

Only answer (a), (b), (c) or (d)"



Structural Break Prompts

"Input:.

Prompt 1: Detection

Question: can you detect any regime switches or structural breaks in this time series? Only answer 'Yes' or 'No'."

Prompt 2: Classification

"Given the following definitions:

Regime Change: A shift in the time series data's statistical properties, such as mean, variance, or auto-correlation, that persists over time. This change is often gradual and represents a new phase or 'regime' in the data.

Structural Break: An abrupt change in the time series data that leads to a new level or trend. This change is typically sudden and can be linked to specific events or shifts in the underlying process.

Examine the provided time series data and select the correct option:

(a) The time series data exhibits a Regime Change. (b) The time series data exhibits a Structural Break.

Provide your answer as either (a) or (b)."




Volatility Prompts

"Input:.

Prompt 1: Detection

Question: can you detect any volatility in this time series? Only answer 'Yes' or 'No'."

Prompt 2: Classification

"Given the following definitions:

    Constant Volatility: The degree of variation in the time series remains consistent and predictable over time.

    Variable Volatility: The level of variation in the time series changes unpredictably over time, without a clear pattern or structure. 

        Clustered Volatility: The time series exhibits periods where volatility is significantly higher or lower, with these periods tending to cluster together.

        Leverage Effect: The volatility of the time series tends to increase when the series experiences negative returns, reflecting an asymmetric response to negative versus positive market movements.

        Select one of the following answers: 

        (a) The time series has constant volatility, (b) The time series has variable volatility, 
        (c) The time series has clustered volatility, (d) The time series has leverage effect volatility 

        Only answer (a), (b), (c) or (d)"





§ LICENSES

Table <ref> lists the licenses for the assets used in the paper.


