Zero-shot and few-shot time series forecasting with ordinal regression recurrent neural networks
Bernardo Pérez Orozco Stephen J Roberts
cs.LG stat.ML












Zero-shot and few-shot time series forecasting with ordinal regression recurrent neural networks
    Bernardo Pérez Orozco^1,2 and Stephen J. Roberts^1,2


We acknowledge the support provided by CONACYT (CVU #598304) and the UK Royal Academy of Engineering.







1- Information Engineering, University of Oxford



2 - Mind Foundry Ltd. 

    December 30, 2023
===================================================================================================================================================================================================================================================




Recurrent neural networks (RNNs) are state-of-the-art in several sequential learning tasks, but they often require considerable amounts of data to generalise well. For many time series forecasting (TSF) tasks, only a few dozens of observations may be available at training time, which restricts use of this class of models. We propose a novel RNN-based model that directly addresses this problem by learning a shared feature embedding over the space of many quantised time series. We show how this enables our RNN framework to accurately and reliably forecast unseen time series, even when there is little to no training data available.




	
	

§ INTRODUCTION


Deep and recurrent neural networks (RNNs) have achieved great success across a wide range of sequential learning tasks, such as machine translation, speech recognition and time series forecasting (TSF) <cit.>. However, RNNs are ultra-parameterised models that overfit without sufficient data at hand, making them unfit to solve  tasks where few data are available. We propose a novel RNN approach to infer full predictive distributions for TSF tasks where there is little to no training data at hand, respectively referred to as few-shot and zero-shot learning. We achieve this by learning a shared feature representation across multiple forecasting tasks, then transferring this  knowledge to help predict unseen time series, even in observation poor environments. We show that our approach outperforms other (widely used) approaches across a range of metrics as well as providing a framework to visualise and interpret the embedding learnt by the neural network. 

Related work. There is a rich literature associated with transfer learning and few-shot learning, especially in the Computer Vision community, where pre-trained models such as VGGNet, ImageNet and AlexNet are regularly used as either transferrable feature extractors, followed by a simpler classifiers, or as initial priors before fine-tuning a model.


However, to the best of the authors' knowledge, the transfer learning literature for time series forecasting is significantly more scarce and often limited to evaluations solely based on the residuals of point forecasts. Full probabilistic forecasts are, however, desirable as this enables practitioners to incorporate predictive uncertainty in their decision-making processes. This is often achieved using Gaussian Processes (GPs) <cit.>, non-parametric probabilistic models that are known to generalise well even with little data.






























































































	
	
	
	

§ METHODOLOGY



Our strategy consists in inferring a joint Memory-endowed Ordinal Regression Deep Neural Network (MOrdReD) <cit.> over a collection of (auxiliary) time series. This results in a cross-task feature embedding that we use as a basis to forecast unseen time series even with scarce data. We briefly summarise this model below, but refer the reader to <cit.> for a more detailed explanation. 



 §.§ Memory-endowed Ordinal Regression Deep Neural Networks

MOrdReD is an ordinal regression approach to time series forecasting that has been shown to achieve long-term, reliable performance. In the ordinal setting, real-valued time series observations are firstly discretised into M (normally equal-sized) subintervals 𝒞 = {𝒞_k}_k=1^M and encoded using a 1-of-M scheme. Such discrete symbol sequences have been most efficiently dealt with in NLP tasks by deep recurrent neural networks and in particular sequence-to-sequence extensions of the Long Short-Term Memory (LSTM) model <cit.>.

Sequence-to-sequence models fit together two LSTM layers, an encoder f^(enc) and a decoder f^(dec). In the first stage of the model's forward pass the encoder sequentially scans P measurements of the time series 𝐗^(τ-1) = (𝐱_τ - P,…,𝐱_τ - 1) and summarises them as temporal characteristics 𝐡^(dec)_0, 𝐂^(dec)_0. These are then fed as initial states to the LSTM decoder, which iteratively produces an output sequence. The (t+1)-th element of the output sequence, 𝐱̂_t+1, is then given by:

    (𝐡^(dec)_0, 𝐂^(dec)_0)    = f^(enc)(𝐗^(τ-1)),
    𝐱̂_t+1   = Softmax( f^(dec)(𝐡^(dec)_t-1, 𝐂^(dec)_t-1, 𝐱_t)).

The softmax output 𝐱̂_t+1 can be interpreted as both a categorical distribution over the set of ordinal bins, 𝒞 = {𝒞_k}_k=1^M, at time t+1, and as a piece-wise uniform over the original time series range. Output sequences are computed by iteratively feeding back 𝐱̂_t+1 into the model and predictive posterior distributions can readily be approximated through Monte Carlo Dropout <cit.>.




 §.§ A General Unified Model for time series forecasting

Our GUM approach is based on inferring a shared representation space of time series. This shared embedding is learnt by fitting a joint MOrdReD model over a single compilation dataset 𝒳_aux of 20 different auxiliary time series which are assumed to possess sufficient observations for training. The datasets were drawn from a variety of application domains, and further detail is given in <https://github.com/bperezorozco/ordinal_tsf>. 

Crucially, our ordinal regression cross-task embedding is learnt over symbolic patterns that are invariant to the original scale of the time series, as all auxiliary time series are quantised independently, but with the same number of bins M=150. A variety of shapes and motifs from the auxiliary datasets are thus encoded in a feature space of ordinal bin motifs. We analyse this cross-task embedding in Section <ref>. 

We note that the quantiser proposed in <cit.> requires prior knowledge of the time series range. With scarce data, the observed range could change within short predictive horizons. We handle this with a simple linear heuristic that estimates the range boundaries within a predictive horizon P_h. Given the largest first-order finite difference of the time series, Δ_max, our linear heuristic approximates the boundaries 𝐗̃_min, 𝐗̃_max = 𝐗_min -P_hΔ_max, 𝐗_max +P_hΔ_max. Future work for improving this estimation includes relating it to Lipschitz constant inference  <cit.>. 














































	
	













	
	

§ EXPERIMENTS






 §.§ Task 1: Zero-shot forecasting

We first assess the model's performance at zero-shot time series forecasting. 

Data. We used two dataset collections in our experiments. The first was used to fit our GUM ordinal regression neural network, and contains 20 varied auxiliary datasets drawn from <cit.> with outputs evenly quantised over M=150 bins. A copy of our auxiliary datasets is provided in <https://github.com/bperezorozco/ordinal_tsf>. The second collection is our test data, which corresponds to the yearly-frequency segment of the M4 competition dataset <cit.>. We drew 150 excerpts that had no missing data (no interpolation needed) and that had at least 36 observations available.

Task. Every M4 time series was split into T=21 timesteps, assumed to be observed, and P_h=15 timesteps for prediction. For our baselines, each segment represents the training and test data, respectively; for GUM, the quantised 21-sample excerpt is fed as the encoder input sequence, before the decoder computes the forecast with horizon P_h=15. Importantly, no neural network parameters are updated whilst scanning the 21-sample sequence.

Baselines. We compared our GUM neural network against two popular models: Gaussian Process Regression (GPR) <cit.> and state-space AR(p) models <cit.>. Detailed derivations of both methods can be found in the given references. In the case of GPR, we fitted two instances for every time series: one with the Mátern 5/2 kernel and one with the Rational Quadratic kernel, both commonly found in the literature to model a wide range of physical phenomena. In the case of State-space AR(p), we fitted two instances with lookback p=3,4. 





Metrics. We evaluated models using three metrics: (1) the Negative log-likelihood (NLL): through this we assess how well  predictive densities capture the ground truth prediction; (2) The root mean squared error (RMSE): to measure the mean predictive accuracy of the models; and (3) Quantile-Quantile (QQ) distance: QQ plots are graphical tools used to assess how well predictive densities match empirical probabilities. We summarise how well calibrated predictive distributions are through the QQ distance, defined as the deviation between the identity and the obtained QQ plot. We refer the reader to <cit.> for full details.

Results. We present our results in Table <ref>. The Table is divided into two sections. On the left side, we show the percentage of datasets in which our GUM neural network outperforms each of our benchmarks for every metric. We show that across all metrics, our model consistently performs better than all baselines for the majority of cases. On the right side of Table <ref>, we provide the average rank of our models. Ranks from 1 (best) to 3 (worst) were assigned for every metric, dataset and model and averaged across all M4 evaluation datasets. In the case of GPR and AR(p), the instance with the best performance across kernels or model order p was chosen. This metric shows that, on average, we expect GUM to outperform all other models; and additionally, when it does not, it can still be expected to come as a second competitor.




 §.§ Cross-task feature embedding analysis

We now provide a visual analysis of the cross-task feature embedding learnt by GUM. We first compute 𝐡^(dec)_0 (as described in Section <ref>) for a random sample of auxiliary time series excerpts, in addition to all M4 evaluation input sequences. Intra-group minimum variance clusters are then computed through Ward's agglomerative clustering. Results in Figures <ref> and <ref> are provided for K=36 clusters, which yields the largest silhouette score for values between 5 and 50. 

We show the resulting frame clusters in Figure <ref>, observing that those in red are only seen by our GUM network at testing time. We note that the excerpts mainly group towards the right-hand edge of each subplot, i.e. every cluster contains patterns with diverse features (such as degree of smoothness, number of optima and rate of change) that fire similar activations, which then produce similar forecasts at the output layer. 

In Figure <ref> we show the 2D UMAP <cit.> representation of each frame's feature vector 𝐡^(dec)_0, colour-coded by cluster id. Bluer and redder markers represent time series clusters that converge to lower- and higher-valued bins at the last timestep, respectively. The smooth colour degradation from left to right, with blue clusters grouping at the left-hand edge and red clusters towards the right, suggests that our model is learning an approximate ordering over ordinal bins.

 





 §.§ Task 2: Few-shot forecasting

We now assess model performance in the few-shot setting.

Data. We use the same 20 auxiliary datasets from Section <ref> to train GUM. For the testing phase, we now use 10 series of the monthly-frequency segment of the M4 dataset, those with no missing observations and with at least 1500 measurements available for model inference. All datasets are quantised using M=150 bins.

GUM setup. In this setting, we pre-train a number of GUM models over the auxiliary datasets with different hyperparameters: hidden units n_h ∈{64, 128, 256, 512}, dropout rate θ_drop∈{0.25, 0.5} and L_2 regularisation constant λ∈{10^-5, 10^-6, 10^-7, 10^-8}. These serve as initialisation parameters for each M4 time series. Models are trained for up to a further 50 epochs with early stopping and the best hyperparameters for each dataset are chosen by grid search over a validation segment.

Task, baselines and metrics. We forecast P_h=100 measurements ahead from a lookback window of P=50, using the same baselines and metrics from Section <ref>, in addition to a naive MOrdReD baseline that has not been pre-trained, i.e. is initialised from scratch and allowed twice as many epochs, also with early stopping. The AR(p) baseline's lookback is chosen from p∈{3, 6, 12, 18}.

Results. We note that our GUM neural network consistently outperforms our baselines across all metrics, as shown in Table <ref>. Crucially, GUM also outperforms the naive MOrdReD baseline across all metrics, suggesting that transferring knowledge from prior tasks makes a substantial contribution to predictive performance.



























	
	

§ CONCLUDING REMARKS


We here introduce a novel neural network model, able to infer probabilistic time series forecasts in both zero- and few-shot learning settings. By introducing a quantisation scheme, we show how knowledge learnt from auxiliary time series can be transferred to previously unseen time series with scarce data. We provide empirical evidence that our approach performs well across a range of metrics, in comparison to other popular methods. By examining the neural activations of our model, we also show how our cross-task feature space encapsulates concepts such as ordinality and knowledge transfer.












	
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	
