Spatiotemporal Adaptive Neural Network for Long-term Forecasting of Financial Time Series
Philippe Chatigny Jean-Marc Patenaude Shengrui Wang
cs.LG stat.ML



#1#2#1#22mu#1#2


























label1]Philippe Chatigny
Philippe.Chatigny@usherbrooke.ca
label2]Jean-Marc Patenaude
jeanmarc@laplaceinsights.com
label1]Shengrui Wang
Shengrui.Wang@usherbrooke.ca
[label1]Department of Computer Science, Université de Sherbrooke, Sherbrooke, Québec, Canada
[label2]Laplace Insights, Sherbrooke, Québec Canada


Optimal decision-making in social settings is often based on forecasts from time series (TS) data. Recently, several approaches using deep neural networks (DNNs) such as recurrent neural networks (RNNs) have been introduced for TS forecasting and have shown promising results. However, the applicability of these approaches is being questioned for TS settings where there is a lack of quality training data and where the TS to forecast exhibit complex behaviors. Examples of such settings include financial TS forecasting, where producing accurate and consistent long-term forecasts is notoriously difficult. In this work, we investigate whether DNN-based models can be used to forecast these TS conjointly by learning a joint representation of the series instead of computing the forecast from the raw time-series representations. To this end, we make use of the dynamic factor graph (DFG) to build a multivariate autoregressive model. We investigate a common limitation of RNNs that rely on the DFG framework and propose a novel variable-length attention-based mechanism (ACTM) to address it. With ACTM, it is possible to vary the autoregressive order of a TS model over time and model a larger set of probability distributions than with previous approaches.  Using this mechanism, we propose a self-supervised DNN architecture for multivariate TS forecasting that learns and takes advantage of the relationships between them. We test our model on two datasets covering 19 years of investment fund activities. Our experimental results show that the proposed approach significantly outperforms typical DNN-based and statistical models at forecasting the 21-day price trajectory. We point out how improving forecasting accuracy and knowing which forecaster to use can improve the excess return of autonomous trading strategies.




Time Series Forecasting Semi-supervised Learning Dynamic Factor Graphs Neural Networks 







§ INTRODUCTION



In recent decades, DNNs have improved TS forecast accuracy in various social settings  <cit.>. Besides their ability to handle non-linear processes, they provide a cost-effective approach for uncovering relations between TS. DNNs are based on the dynamic factor graph (DFG) framework <cit.>, which is a particular case of a factor graph <cit.> in which the template method <cit.> is applied. Specifically, a DNN-based model assumes that the factors of a DFG are individual neural networks (NN) that enforce a hierarchical structure for pattern detectors throughout its hidden layers <cit.>. Under this framework, the DNN-based model learns complex probability distributions and increases forecast accuracy on mostly homogeneous datasets containing multiple measurements, as well as in applications where there are exogenous variables that are strongly related to the variable(s) of interest <cit.>: e.g., traffic <cit.> or electricity load forecasting <cit.>.

However, training a DNN remains difficult for most TS settings <cit.>, especially when TS are non-ergodic, heteroskedastic, non-stationary or have high noise-to-signal ratios. Such cases are often found in financial TS. Few DNN-based models have demonstrated consistent accuracy on such datasets spanning multiple years for different asset classes <cit.>. Besides reasons associated with concept drift <cit.>, the difficulties in forecasting financial TS are also due to the fact that most DNN learning frameworks do not appear to be adapted for this setting. Training a DNN needs a large dataset of independent training samples that are representative of the data to infer. Aside from applications like intra-day forecasting <cit.>, most financial applications rely on TS that have a relatively limited number of measurements <cit.>. Additionally, historical price trajectories can be very noisy and their behaviors exhibit complex cyclical effects <cit.>. As it is not possible to obtain multiple independent realizations of a specific asset's price fluctuation under different circumstances for the same time period <cit.>, the nature of financial TS necessarily leads to both a lack of training data and the well-known difficulty in modeling their long-term effects <cit.>.

This paper proposes a more effective DNN framework for forecasting multiple financial assets conjointly and enhancing the capability of the TS model to learn a larger set of probability distributions. The key contributions of this paper are as follows:

 
  * We propose a novel attention mechanism for the Dynamic Factor Graph (DFG) framework. This mechanism offers the capacity to consider a variable number of past latent states over time.
 
 
  * We make use of the novel attention mechanism to optimize the order of an autoregressive (AR) generative function over time. We show how such a mechanism can model non-stationary distributions while keeping a constant parameterization.
 
 
  * By incorporating the attention mechanism, we develop an energy-based deep generative approach for modeling interactions between multiple TS to generate multivariate forecasts. Our spatiotemporal adaptive neural network (STANN) is able to operate under a limited data constraint by exploiting prior knowledge of the TS to "virtually" augment its training samples and allows the discovery of interrelations between TS.

 
  * We have conducted an extensive experimental evaluation showing the effectiveness of the proposed model for forecasting 21 daily return trajectories of exchange-traded funds (ETFs) and mutual funds (MFs). We also show preliminary but promising results of the proposed model for improving autonomous trading strategies. Of all the models proposed in the last 10 years <cit.>, ours is, to our knowledge, the first to outperform naive baselines in a monthly multivariate financial TS setting.
 

The remainder of this paper is organized as follows: Section <ref> reviews major existing work on modeling TS in social settings and relevant notions related to the DFG. In Section <ref>, we present our model and describe its training procedure. In Section <ref>, we present the setup of our empirical evaluation, which extends over more than 19 years of financial market activities, and describe our results. Section <ref> presents our conclusion.



§ RELATED WORK





 §.§ Prior Work:


Different formulations <cit.> of DNN models have been introduced to facilitate their application on TS data. While promising results have been achieved recently for financial TS prediction, as in <cit.>, it has been pointed out that much of the published machine learning (ML) work in the TS literature claims satisfactory accuracy without adequately comparing the methods used against conventional methods <cit.> and, further, that it relies on inappropriate criteria <cit.>. In fact, only a few authors, such as <cit.>, have been able to show that their models yield better performance on multiple TS than simple statistical models like ARIMA or even a naive forecast. Most work uses non-scaled error metrics to assess forecast quality on multiple TS. However, it has been known for years that comparing forecasts of multiple TS of different scales via non-scaled metrics often leads to misleading results <cit.>. The myriad of existing DNN-based models <cit.> applied to financial settings and the results presented around them have raised undue expectations that such methodologies provide accurate predictions at forecasting multiple TS, while there is clearly a lack of experimental demonstration that they outperform simple baselines in the majority of cases.

Nonetheless, large gains can still be achieved by using DNN and ML approaches. Recently, state-of-the-art accuracy was achieved at the M4 competition <cit.>, where the top 2 entries used DNN-based or ML techniques along with statistical models. Subsequent to these findings, the authors in <cit.> were the first to show that it was possible to build a pure DNN-based model for this task and achieve greater gains than the best competition entry <cit.>. Given the wide range of TS to forecast[The M4 dataset contained 100,000 individual TS, of which approximately 25% were financial TS of different types.], the top-performing models submitted relied on ensemble techniques to be robust over the different types of series.

Direct comparison between single and ensemble models is generally unfair, as ensemble models permit the modeling of various probability distributions using multiple TS models and subsequently apply some form of forecast combination by evaluating the inference capabilities of each TS model a posteriori. However, the findings from these models can be investigated with a view to building better individual models. For instance, the DNN-based models, which performed well on this dataset <cit.>, provide insights into techniques that can be used to improve the performance of individual models: e.g., residual connections between hidden layers, adaptive learning rate scheduling, input preprocessing and both seasonal and trend decomposition embedded directly in the model. Most of these techniques are "tricks" to facilitate DNN learning. However, the idea of applying a signal decomposition within a neural network is promising and several authors <cit.> have shown its effectiveness on real-world datasets. Given the well-known difficulty of dealing with the raw signal of financial TS, we raise the question whether a better representation of these TS can be learned directly by applying such decomposition within the learned latent variables of a DFG <cit.>.



 §.§ Dynamic Factor Graph:



A DFG consists of an undirected acyclic state-space model where factors are replicated on a fixed time interval T={t_1, ..., t_T} to model a probability distribution. A DFG models the joint probability P(X, Z;W) of the observable values X={x_1, ..., x_T} and the latent variables Z={z_1, ..., z_T} given some parameterization of all factors in the graph W as in Eq. <ref>. Here, ℒ is the partition function and E(X,Z;W) ∝ - logP(X,Z|W) + const is the total energy of the model. The total energy of the model is the sum of the normalized probability scalars assigned by a factor to all possible input data points associated with it. To obtain a probability, the total energy is normalized by ℒ.


    P(X, Z;W) = e^-β E(X,Z;W)/∫_X'∫_Z'e^-β E(x',z';W)d x' d z'  = e^-β E(X,Z;W)/ℒ


    E(X,Z;W) = ∑_t∈ T∑_F ∈ℱ E(A_t, O_t; F); A_t∈ Z, O_t∈{X, Z}



    E(A_t, O_t; F)  = 
             error(g(Z_t, W_g), Z_t+1)     if  F=g
    
             error(d(Z_t, W_d), X_t)     if  F=d
             
             ; ℱ={d,g}


The energy term for a given sequence of observable values X and latent states Z is given by Eq. <ref>, with the energy term of a single factor defined by Eq. <ref>. Here we assume that our DFG follows a parameterization similar to that of an HMM architecture of order 1 with two factors, i.e., ℱ={d,g}, and W_F is the parameterization of the factor F∈ℱ. The higher the energy term between an input data point and its associated output data point, the less probable it is that the value will be observed. Despite the fact that the DFG's edges are undirected, the energy term of each factor is not. Hence, training a DFG for TS forecasting is similar to adjusting the parameters of a Dynamic Bayesian Network (DBN) <cit.> where we simply need to adjust the parameters of the factors using maximum likelihood estimation, which is equivalent to reducing the total energy of the model.

For our particular case, where we consider an HMM under the DFG framework, the main difference between an RNN and this particular DFG is how the state-space component is used. However, since Eq. <ref> is intractable for continuous variables under non-Gaussian distributions, we estimate the mode of the distribution instead by maximum a posteriori approximation <cit.>. Thus, an HMM-based DFG model learns the probability distribution ℙ of a TS using two factors replicated over time: a decoder factor and a dynamic factor, i.e. ℱ={d, g}. The decoder factor d(Z;W_d) is a function that models the maximum likelihood of observing a random variable X_t given latent variable Z_t: 


    X̃_̃t̃ = d(Z_t, W_d) ≙max_xℒ(x|Z_t=z_i; W_d); z_i ∈ Z, x ∈ X


with W_d being the parameterization of the factor, t a particular time point and ℒ the likelihood function. The dynamic factor g(Z;W_g) models the maximum likelihood of observing a state given some prior state and is defined by Eq. <ref>. g(Z;W_g) models a transition probability distribution as in the DBN framework:


    Z_t+1 = g(Z_t, W_g) ≙max_zℒ(Z_t+1=z|Z_t=z_j, ⋯, Z_t-k=z_j');
     z, z_j, z_j'∈ Z


Note that one must specify the order of g(Z;W_g) by changing its configuration: Z_t+1 = g(Z_t, ..., Z_t-k; W_g)), where k is the autoregressive (AR) order of the process. Doing so makes the assumption that the probability distribution ℙ models a stationary process if, for all t, ℒ(Z_t+1=z_i|Z_t-k:t=z_j)≥ 0 and is constant for all t <cit.>[Here, Z_t-k:t corresponds to all Z_t'  included between Z_t-k and Z_t, where t'∈ T]. This assumption holds for both the discrete and the continuous case <cit.>. When modeling ℙ using a graphical model, we use the notation ℙ (A B | C) <cit.> to indicate that ℙ models a local independence relation between a set of nodes A and B given C. In particular, assuming that the probability distribution models a stationary process induces the following set of independence relations:

    
  * The latent variable evolves in a Markovian or a semi-Markovian way:
    
    ℙ (Z_t+1 Z_0:t-1-k| Z_t-k:t)

    
  * The observation variables at time t are conditionally independent of the state sequence given the latest k+1 state variables at time t <cit.>: 
    
    ℙ (X_t Z_0:t-1-k, Z_0:t+1:∞| Z_t-k:t)

 

This AR order is a hyperparameter that needs to be tuned carefully, since the true probability distribution is intractable in most cases <cit.> and the set of local independences cannot be verified in practice. Assuming that the AR parameters are constant can impair training, as the resulting AR weights are optimized to reduce the average error. This limitation is problematic if the AR order was not selected appropriately or the training data contains multiple TS dynamics. In this work, we address these limitations by proposing an attention mechanism that enables a DFG to select its AR order automatically and adjust it over time. The stationary assumption can thus be relaxed such that the set of interdependences in Eq.<ref> and Eq.<ref> holds but the process order k is a function of time. This permits non-stationary probability distributions to be modeled since ℒ(Z_t+1=z_i|Z_t-k:t; W_g)≥ 0 but is not necessarily constant over time.





§ THE STANN MODEL





 §.§ Model Definition Without Including Time Series Dependencies

Given X: ℝ^T × n × m, a 3-dimensional tensor representing a set of n TS of length T and dimensionality m, we define X_t, i, j as the value of dimension j for TS i at time t. The task of interest is to predict n multivariate TS τ time steps ahead X̃: ℝ^τ× n × m. We represent the spatial relationship between series within a 3-dimensional tensor, that we denoted by W: ℝ^n × R × n, where R is the number of relations considered. Thus, our aim is to train a model f: ℝ^T × n × m + [ℝ^n × R × n] →ℝ^τ× n × m. 

In STANN, we use a particular formulation of the DFG. The decoder factor d(Z; W_d) decodes the expected variation between X_t-1 and  X_t from the latent factor Z_t, which allows the decoder to be defined as in Eq. <ref>. Here X̃_̃t̃ is the prediction computed at time t.


    X̃_t = X_t-1 + d(Z_t; W_d)
 

The dynamical module g(Z; W_g) is defined by Eq. <ref> and considers the past k+1 relevant latent factors Z_t - k: t, i.e. Z_t - k to Z_t. d(Z;W_d) and g(Z;W_g) is implemented as a doubly residual stacking NN, as in N-BEATS <cit.>. In contrast to N-BEATS, we apply the TS decomposition on the latent factors rather than the raw signals.


    Z̃_t+1 = g(Z_t - k:t; W_g)




 §.§ Adaptive Computation Time for Autoregressive Order Selection



As mentioned in the previous section, assuming that the forecast depends on a fixed AR order covering the past k observations is a strong assumption that can impair model training if the autoregressive order is not selected correctly. RNNs, like the long short-term memory (LSTM) nework <cit.>, consider the past k observations by maintaining in memory a state vector that allows them to retain information as long as required and forget it when it is no longer relevant. Unlike LSTM, we permit k to vary adaptively without requiring the observable value as input. To this end, we propose an adaptive attention-based mechanism to enable DFG to be memory-augmented. Our attention mechanism is inspired by the Adaptive Computation Time (ACT) algorithm proposed in <cit.>, denoted as actm(Z; W_actm) in our model. actm(Z; W_actm) is a factor that generates a probability distribution on Z that is used to select the order of the regression in Z_t - k:t. The probability function associated with the factor at each time t is modeled by a parametric function f_actm(Z_t; W_actm), with W_actm being its parameterization. For ease of computation, the order selection and the computation of Z_t - k: t are performed by using the sum of a certain number of past latent factors weighted by probability distribution function f_actm as in Eq. <ref>. 



    Z_t-k:t=actm(Z_t, W_actm)=∑_0≤ k'<k:[b(t)>ϵ]φ_Z_t-k'Z_t-k'


    φ_Z_t-k' =   f_actm(Z_t-k', W_actm)    if  b(t)-f_actm(Z_t-k', W_actm) > ϵ
    
         b_   if   b(t)-f_actm(Z_t-k', W_actm) ≤ϵ



Specifically, actm(Z;W_actm) is implemented with two budgets b(t) = {b_=t,b_=1}: one to keep account of available past time steps and one to track the cost of considering a latent factor. We start with the current latent state factor Z_t and consider whether to include Z_t-k' for k'=0, 1, 2, ....  Each time we consider a latent factor Z_t-k', we reduce our budget b_ by 1 and b_ by φ_Z_t-k'=f_actm(Z_t-k', W_actm), the latter being bounded within ]0, 1[. If a budget goes below ϵ, i.e., either b_<κ or b_=0, we stop considering any more latent factors and attribute the remaining cost budget to the last factor considered. κ∈ℝ^+ is a small constant (0.01 for the experiments in this paper), whose purpose is to allow the selection of an AR(1) process.

Hence Eq. <ref> can be reformulated as Eq. <ref>.

    Z̃_t+1 = g(Z_t - k:t; W_g, W_actm)

We can interpret actm(Z; W_actm)'s objective as evaluating the quality of each past latent factor and assigning the appropriate autoregressive  weight at times t-k' that maximizes the log likelihood of the generative process modeled by Eq. <ref>. Since actm(Z; W_actm) uses b_ to determine how many past steps are available, we can theoretically account for all previous learned factors if ∑_k=0^t f_actm(Z_t-k, W_actm) < 1-κ. Note that the imposed budget restricts each autoregressive weight to be between 0 and 1, with the sums of all the weights being equal to 1. We apply this mechanism solely within g(Z; W_g) to facilitate the training model, but the approach could also be extended to d(Z;W_d). From now on, then, we will simplify our notation by considering that W_g also includes the ACTM parameters. The attention mechanism is summarized in Figure <ref> and can be designed as any configuration of a feedforward network with a sigmoid activation function. An illustration of our model with ACTM is presented in Fig. <ref>.





The training procedure consists of minimizing the following bi-objectives loss function (<ref>):
    
    
    
    Loss(d, g, Z) = + 1T∑_t=1^T-1Δ(X_t-1 + d(Z_t; W_d), X_t)

    
    + λ1T∑_t=1^T-1 || Z_t+1 - g(Z_t - k:t; W_g))||^2

    
    
The first term (<ref>) measures the ability of the model to reconstruct X_t from Z_t. The second term (<ref>) measures the system’s capacity to capture the dynamicity of the equation by its ability to link states of Z in sequential order. Δ is a loss function that measures the difference between the prediction X̃_̃t̃ and the ground truth X_t. The second term in Eq. <ref> forces the model to learn latent factors Z_t+1 that are as close as possible to g(Z_t - k:t, W_g), with the ideal case being a solution where Z_t+1 = g(Z_t - k:t, W_g). However, this solution is not valid for the first term in Eq. <ref>, where the ideal case is a solution for which X_t+1 = d(Z_t - k:t, W_d). To balance the relative importance of the two terms, the hyperparameter λ is introduced to reduce or increase the importance of the second term relative to the first term in Eq. <ref>. Training the model using Eq. <ref>[In our experiments, λ was fixed trough a hyperparameter optimization that seach the optimal value in the following interval [0.01 and 1.0]] can be accomplished using any expectation-minimization-based approach <cit.> or an end-to-end <cit.> approach that trains the three factors conjointly.



 §.§ Model Definition Including Time Series Dependencies



Let us now introduce the way interrelations between TS are captured. As pointed out in <cit.>, multiple types of relations between financial TS have been uncovered. To test whether this prior knowledge has predictive capability, we propose that the relationships between the dynamic processes of multiple TS be given as additional prior inputs W ∈ℝ_+^n × R × n to the model, as in <cit.>. We will first formalize how relationships between series are incorporated into the model and how this allows us, "virtually", to have a high number of training samples. Then, we will describe two extensions of this approach. The first extension allows the strength of these relations to be weighted, and the second allows the model to learn these relations directly without any prior information.

Relationships between the dynamic processes of n TS are incorporated via a tensor W ∈ℝ_+^n × R × n, where R is the number of relation types given as prior.In the following discussion, each sequence is indexed by X_t,i, while the corresponding hidden state is represented by Z_t,i. X_t,i correspond to the particular observation of the iiem TS at time t and Z_t,i is its corresponding hidden state. We formulate that, at time t, Z_t+1,i depends on its own latent representation (intradependency) and on the representations of other series (interdependency).

Intradependency is modeled through a linear mapping Θ^(0)∈ℝ^n × n. Interdependency is modeled with one transition matrix Θ^(r)∈ℝ^n× n for each possible type of relation r ∈ R. Θ^(r) learns the relationship between each TS by applying a linear combination between neighboring TS and we denote Θ^(r)_i as the linear combination learned for relation r of the iiem series. We denote all the transition matrices by Θ^(R)∈ℝ^R × n × n and W_i^(r) as the relation given as prior between the iiem series and other neighboring TS. To evaluate Z_t+1, we compute the matrix product between the latent space Z_t and its dependencies (Θ^(0), Θ^(R)) as in  Eq. <ref>. The decoder follows along, using Z_t,i as inputs, and computes the expected variation as in Eq. <ref>. h_g, h_d are the respective activation functions of g(Z;W_g) and d(Z;W_g).


    Z̃_t+1, i =     g(Z_t-k:t, i; W_g, Θ)
      =    h_g(actm_g(Z_tΘ_i^(0)  + ∑_r ∈ R W_i^(r) Z_tΘ_i^(r)); W_g)


    X̃_t+1,i = X_t,i + d(Z_t,i; W_d) = h_d(Z_t, i; W_d)


Note that Z_t is shared among all series with respect to g(Z; W_g), but the representation of each series is disentangled explicitly by means of W; i.e., d(Z;W_d) takes as input Z_t,i, the hidden factor of the ith TS. Doing this has two advantages:
[label=(*)]

  * g(Z;W_g) can forecast Z̃_t+1 with fewer regressors.

  * It "virtually" increases the number of training samples, as we can use time and positional coordinates to make T × n fixed-size training samples instead of handling TS as sequential data. With respect to each disentangled latent state, the correlation existing between the latent states of two TS would indicate that our model estimates that the TS follows similar trajectories despite W specifying that they are or are not correlated which is possible in part due to Θ^(R).




 §.§ Model Extensions



The two possible extensions proposed in <cit.> can also be applied to our model. We summarize the extensions here; readers are invited to refer to the original paper <cit.> for a more detailed explanation. The first extension, denoted by , consists of adding a learned matrix of weights Γ^r∈ℝ_+^n × n that can reduce the strength of relations given as prior. The second extension, denoted by , consists of replacing W with Γ such that the model learns both the relational structure and the relation weights within Γ. Applying the  or  extension formalizes Eq. <ref> as in Eq. <ref> or Eq. <ref>, respectively, where ⊙ signifies element-wise multiplication between two matrices:


    Z_t+1, i = g(Z_t-k:t, i; W_g, Θ) = g( actm_g(Z_tΘ_i^(0) + ∑_r ∈ R (Γ_i^(r)⊙ W_i^(r)) Z_tΘ_i^(r))



    Z_t+1, i = g(Z_t-k:t, i;W_g, Θ) = g( actm_g(Z_tΘ_i^(0) + ∑_r ∈ RΓ_i^(r)Θ_i^(r))


The optimization problem can thus be adjusted for Γ, depending on whether the dynamic function is specified by Eq. <ref> or Eq. <ref>, and can be written as Eq. <ref>. |Γ| is a l_1 regularizing term intended to sparsify Γ^(r); γ is a hyperparameter set to tune this term; and λ is a factor set to balance the relative importance of g(Z; W_g) and d(Z;W_d).


    d^* g^*, actm_g^*, Θ^*, Γ^* = *argmin_d, Z, Γ,1T∑_t Δ(d(Z_t; W_d) + X_t-1, X_t) 
     + γ|Γ| + λ1T∑_t=1^T-1 || Z_t+1 - g(Z_t-k:t; W_g, Θ)||^2




§ EXPERIMENTS





 §.§ Datasets and Experimentation Procedure





We report here the results of an experimental evaluation of our forecasting methods on two datasets: 𝒟_1= and 𝒟_2=. The two datasets, summarized in Table <ref>, were obtained through FastTrack[<https://investorsfasttrack.com>]. They were selected for restraining the number of training samples and as representing respectively a low-data setting and a medium-data setting. Both datasets contain daily closing prices of U.S. MFs and ETFs traded on U.S. financial markets, each covering different types of asset classes including stocks, bonds, commodities, currencies and market indexes, or a proxy for a market index. Taken in combination, they cover 19 years of financial market activities and provide an overall view of the whole financial ecosystem. Each TS of these datasets represents the aggregation of multiple individual financial assets. In some of these TS, like , the aggregation of these individual TS is subject to vary over time with respect to management activities associated with these funds.



    MASE(X̃, X) = 1/H∑_i=1^H|x_T+i - x̃_T+i|/1/T+H-s∑_j=s+1^T+H|x_j - x_j-m|



    THEILU(X̃, X) = 1/H∑_i=1^H√(|x_T+i - x̃_T+i|^2)/1/T+H-s∑_j=s+1^T+H√(|x_T+i - x̃_T+i|^2)



    SDILATE(X̃, X, X̃'̃) = (α) loss_shape(X̃, X) + (1-α) loss_time(X̃, X)/(α) loss_shape(X̃', X) + (1-α) loss_time(X̃', X)



    MDA(X̃, X) = 1/N∑_i=0^τ sign(X̃_t:t+i-X_t-1)=sign(X_t:t+i-X_t-1)


To train each model, we carried out an evaluation on a rolling-forecasting-origin cross-validation, setting the number of time steps τ to 21 days for simulating forecasting on a monthly basis. All models were trained on normalized TS using the interquartile range method. Produced forecasts were unscaled back to the original TS scales to measure the forecast's error. All DNN-based models were trained using stochastic gradient descent (SGD) with Adam <cit.> and a learning rate scheduler <cit.>. The number of epochs, learning rate and other model hyperparameters, such as the optimal training window or the number of hidden layers and the number of hidden neurons for DNN-based models, were determined by a Bayesian hyperparameter search <cit.>.



 §.§ Time series Models



For fairness of comparison, we considered only models that can forecast multivariate TS directly, with the exception of two baseline models. The models used are as follows:




  * Naive: A simple heuristic that assumes the τ future steps will be the same as the last previously observed.


  * AR: A classical univariate autoregressive process in which each TS is forecasted individually. The prediction is a linear function of past l lags.


  * ARIMA: An autoregressive integrated moving average model that forecasts each TS individually. Implementation of ARIMA was done with <cit.> to automatize the selection of the best parameterization over the training set.


  * LSTM: A long short-term memory model that forecasts τ steps ahead in an iterative fashion <cit.>. LSTM with hidden layers and the number of hidden neurons were considered. 


  * LSTM-A: The same model as LSTM but with an added softmax attention layer to weight the importance of each past latent state for forecasting the next step-ahead. 


  * WaveNet: A convolutional neural network using dilated causal convolutions <cit.>. 


  * 1-BEATS: A member of the neural basis expansion analysis for time series forecasting ensemble model presented in <cit.>.


  * STNN: The closest model to ours. STNN can be considered as a particular case of our model, i.e., our model with k=1. The two extensions of STNN (STNN-R and STNN-D) <cit.> were also considered. The Pearson correlations between TS were computed over the training set to define W. We used the same training strategy as for STANN, i.e., modeling the variation only and training the model end-to-end to establish a fair comparison between model architectures.


  * STANN: The model proposed in this paper. The two extensions presented in Section <ref> were also considered. The extensions expressed in Eq. <ref> and Eq. <ref> are denoted by STANN-R and STANN-D, respectively. Pearson correlation was used to define W.








 §.§ Forecasting Performance

Our experimental results are summarized in Table <ref> and Table <ref>. First, we analyze the average performance of all the models and the statistical significance of the results obtained. Our model outperforms the DNN-based and statistical baselines in terms of all metrics on both datasets. The values of these metrics also indicate the superiority of the training framework of STNN and STANN as compared to the other models evaluated. By using the proposed attention mechanism and the TS decomposition approach of N-BEATS, STANN improves on the performance of its base model (STNN).

The augmentation trick used in the STNN and STANN models, presented in Eq. <ref> and Eq. <ref>, is the largest contributing factor behind these results. By exploiting prior knowledge on the relation of these TS, STANN and STNN enhance their ability to forecast TS by "virtually" increasing the number of training samples despite using a shared latent state like LSTM and Wavenet. These results are very promising, considering that
[label=(*)]
    
  * our approach achieved such results using a relatively small number of TS;
    
  * it was trained solely using historical prices.

It is not surprising that DNN-based models (WAVENET, LSTM, LSTM-A, 1-BEATS) underperform compared to statistical baselines when trained in this setting, given their large parameterization and the small number of training samples at their disposal. Our results show that the augmentation trick of STNN and STANN appears to be a solution to the lack of training samples when such models are trained in a multivariate setting. We point out that, contrary to <cit.>, we did not achieve similar MASE for the one-step-ahead forecast. We observe that during model training, we achieved similar results but the accuracy quickly dropped after the first 5 steps ahead in the first few epochs to yield a better overall forecast on the whole trajectory. Hence, there appears to be a trade-off between short-term forecast accuracy and the longer-term forecast accuracy when optimizing DNN-based models.





We can qualitatively compare our models by plotting the absolute scaled error of the individual point forecast (IPF), i.e., |x_T+i - x̃_T+i|/1/T+H-m∑_j=m+1^T+H|x_j - x_j-m|, for all the TS forecast, and comparing where our model fails. We observe that our approach is relatively consistent at forecasting the trajectory of each asset (Fig. <ref>), but the majority of the residuals appear to occur in an epistemic fashion, i.e., the TS forecasting difficulty varies over time. Our proposed attention mechanism slightly increases the forecast accuracy in these episodes of forecast instability over its base model, which explains the majority of the additional average gain in accuracy. 

By looking at the ability of the evaluated models to predict the trend of the out-of-sample trajectory, we observe in Fig. <ref> that the STANN- and STNN-based models outperform baseline models but the STANN-based models show less variance in their results when compared to the different extensions of the STNN-based models. Given that these models tend to forecast the appropriate trend more accurately, this explains why our proposed framework outperforms other baseline models.



In Fig. <ref>, we illustrate, showing the autoregressive order of a sample of the STANN-R model, that our model generates a dynamic process thanks to its attention mechanism. We can see that the model oscillates between AR(1) and AR(2) processes, although it remains more often at an AR(1) process. Interestingly, we can identify time spans for certain assets that are dominated by an AR(2) process and other regions dominated by an AR(1) process. This phenomenon is similar to what regime switching (RS) models <cit.> enforce as prior when modeling the TS. Our approach differs in that we do not have to specify the number of regimes, nor the AR order a priori. However, we notice that the choice of hyperparameters and model architecture can lead to vastly different results, with instances converging to either stationary or higher-AR-order solutions. Hence, further studies are needed to determine how this attention mechanism permits modeling of regime switches within its latent states. 



Finally, we also performed an ablation study to show the effect that the TS decomposition technique and the attention mechanism have on the model. To this end, we plotted the values of MASE metrics of each step-ahead forecast of model STNN-D and STANN-D with one of the two components removed. The plot, presented in Fig. <ref>, shows the reference model (top left), our proposed model (top right), our model without ACTM (bottom left) and our model without the N-BEATS architecture (bottom right). The plot shows that using the attention mechanism alone without TS decomposition increases the overall forecast error but reduces the error propagation often found in recursive approaches. When combined with the TS decomposition architecture presented in <cit.>, we observe a significant error reduction for the last 14 days of the forecast trajectory. Regardless of the number of steps ahead forecast, our approach is significantly better than its base model at reducing IPF ^****.




 §.§ Added Value for Autonomous Decision Making


We conducted some preliminary studies to demonstrate the utility of the proposed model to help autonomous decision making. Specifically, we show how the improved forecasting accuracy achieved by our model enhances the performance of autonomous trading strategies. The challenges of this question are threefold:
[label=(*)]

  * TS models are not decision systems in themselves.

  * We must rely on a trading strategy that considers forecast as a proxy to compare forecasting models. Given the myriad of strategies that exist, selecting one trading strategy over another is highly subjective and may lead to ambiguous results <cit.>.

  * Each trading strategy has its own sensitivity and some trading strategies bypass the forecasting step by directly generating trade positions <cit.>.

This last point is particularly important, as different trading strategies will produce different allocations depending on the same input data. Consequently, the sensitivity of trading strategies, and the state space on which they operate to establish trade signal plays a significant role in determining the excess return. This results in multiple issues that render it difficult to isolate the exact role that forecast accuracy has on various trading approaches, especially more sophisticated ones. It is necessary to point out that an extensive study of these issues is beyond the scope of this paper. However, we discuss briefly these issues and point out similarities and differences  that our approach has over existing ones.


One of the most popular frameworks for portfolio optimization, i.e., the traditional mean/variance framework <cit.>, is a typical framework where the expected returns and variances are estimated and asset allocation decided based on an optimization procedure that aims to minimize a loss function. This approach is known to produce mixed results when exposed to noisy forecasts <cit.>. Alternatively a simple strategy that generates buy and trade signals like in <cit.> can be made to translate the trajectory predicted in a trading action. The advantage of this approach is that they are model-agnostic, i.e., they do not depend on the model choice to apply the prediction and there exists correlation between having good estimates of the returns and the performance of a trading strategy. However, past profits can vanish very rapidly if the trading strategy makes a bad allocation at a bad time. Even if a TS model's accuracy is good on average, it suffices for the TS forecast to be bad at the wrong moment to lose profits from past months. Hence, the timing of the forecasts plays a significant role in the performance of these strategies.


In comparison, other DNN-based strategies can be used to compare our TS model. These approaches generate directly trade signals within their architecture. This is either done in typical supervised setting like in <cit.> where a model aims to solve classification task with each class corresponding to a trade action. In this setting, a trade action must be labeled for each input data. Alternatively, RL approaches like <cit.> can be applied where a model generates trade actions based on their perceived state of their environment. These methods learn a policy which generates a probability distribution over possible trade actions based on a state space and are trained to maximize a reward function. This state space can be fabricated by a selection of different features as in <cit.>. In this setting, we do not require labels for each input data, but we must specify a reward function for assessing the quality of the action chosen based on its environment. To compare the effect of TS models accuracy using either of these two approaches, TS forecast must be included within their input data.

However, additional issues arise from the use of these methods. As stated in <cit.>, RL approaches can exhibit "slow learning and need a lot of samples to obtain an optimal policy." Similarly, this can also be the case of a DNN-based model depending on the model architecture. Unless we have at our disposal, the historical prediction for each time interval at all time points, using DNN and RL approaches is impractical as the amount of training samples to train these methods is diminished by the pace at which we can produce forecasts. Hence, for all DNN-based methods (LSTM, LSTM-A, WAVENET, STNN and STANN), one must resort to zero-shot forecasting, i.e., without explicit retraining on that target data, to produce these forecasts in a reasonable amount of time to justify the use of RL approaches. This, however, augments exposure to concepts drift effect that occurs when not retraining model <cit.>, which, in turns, tends to lower the average accuracy of the TS model <cit.>. Hence, two additional factors can influence the performance of this approach: (1) sensibility to hyperparameters and model architecture of the trading strategy, (2) the number of training samples at our disposal where a forecast has been made. To prevent the leakage of these additional issues, we restrict our experimental analysis to two traditional approaches, which we describe below. 





In our experiments, we studied and observed relations between the accuracy of the TS forecast and the excess return of a strategy. Results for various financial metrics of two common strategies are presented in Table <ref> on both datasets. Here, we include the annualized Sharpe ratio sharpe=E[r_portfolio - r_risk free])/σ_portfolio×√(𝒮), where r_portfolio is the return of the portfolio, r_riskfree is the return of a risk-free strategy, σ_portfolio is the standard deviation of the excess return of the portfolio and 𝒮=τ/252 is an annualization factor, with 252 being the average number of active market days per year. We also include the maximum drawdown, which is the maximum loss observed from a peak to a trough of a portfolio; the mean return per trade and the total profit after the observation periods. We also present the excess return graph of different portfolio strategies built on the evaluated models for the  dataset in Figs. <ref>,  <ref> and  <ref>.




We used two proxy trading strategies to carry out our evaluation.

    
  * Black-Litterman (BL) strategy: The first is a simple, efficient frontier optimization trading strategy <cit.> based on the BL allocation model <cit.>, in which we maximized the Sharpe ratio. The expected returns are determined by the TS forecasts and implied market returns and the estimated risk was computed respectively by the in-sample returns and covariance.[A long-only portfolio constraint was added to ensure that the allocation weights remain between 0 and 1. In some instances this constraint is too restrictive and the convex optimization used in this approach fails. In such cases, we permit the optimization to consider the short position; i.e., allocation weights remain between -1 and 1, and do not consider the short position in our allocation.]
    
  * Simple: a simple trading strategy where one invests equally in each of the securities that the TS model predicted would increase in price. If the trend signal is negative and assets were allocated, we consider the forecast as a "sell" signal. A positive forecast trend is interpreted as a "buy" signal. The allocation weights were normalized according to the strength of the trend signal using a softmax function.

We compare these strategies with the same "optimal" BL strategy where the expected returns and covariance matrix are known in advance to simulate how a perfect TS model would have performed for this strategy. We also include an equal-weight portfolio where we invest in each asset equally.

Allocations of assets were reconsidered on a 21-day basis to simulate how an autonomous trading strategy would change its portfolio allocation over time depending on the financial context. We used a $10,000 initial portfolio value and bought whole security shares, with the leftover money being at a risk-free rate determined by the 30-day U.S. Treasury bill rate. For the  dataset, we used the U.S. 3-Month Treasury rate rather than the 1-month rate for the risk-free strategy, given that no data for the U.S. 1-Month Treasury rate before 2001-07-31 is available from the public and proprietary data sources at our disposal. Transaction fees were assumed to be null and dividends were accounted for within the adjusted closing price.

It can be seen that perfect predictions yield extremely significant returns for BL strategies on both datasets. However, although LSTM and LSTM-A are poor forecasters with respect to all metrics, the strategy built on them is among the top performers. Identifying the principal cause of this phenomenon is not a trivial matter. However, efficient frontier optimization methods are known to produce mixed results when the forecasts are too noisy <cit.>. At this level of accuracy, none of the TS models evaluated are sufficiently robust by themselves to be considered in a BL strategy. However, we can observe that there are significant gains to be made if one could identify the condition where a TS forecaster helps a trading strategy, as each strategy overperforms the baselines at certain moments.

The performance of the simple strategy shows the importance of having a more accurate TS model for autonomous trading strategies. TS models that forecast more accurately, like the STNN and STANN-based approaches, are among the top performers on both datasets with respect to most metrics, especially when few TS are considered. When more TS are considered (), the simple strategy is too naive to select a meaningful subset of securities and will yield performance similar to an equal-weight portfolio. However, since STNN and STANN-based models forecast the trend more accurately, the maximum drawdowns of the simple strategy based on these models were much smaller compared to the LSTM and LSTM-A strategies, which made allocations over a smaller set of securities at a time. Hence, a simple naive strategy can perform relatively well within a curated set of TS using a better forecaster, but will not scale effectively when the number of assets considered is too large.





§ DISCUSSION


This paper proposed a new self-supervised deep generative model (STANN) for forecasting multivariate TS conjointly, which explicitly models the interactions between TS. We introduced a novel attention-based mechanism that enhances the capability of any RNN based on the DFG framework. We showed how this attention-based mechanism increases the set of probability distributions that can be modeled by permitting modeling of non-stationary distributions. Incorporating this into our model, we presented one general approach and two extensions for considering interrelations between TS. We showed that when these interrelations are incorporated, we can fit these DNN-based models even where little training data exists. Experiments were performed on two financial datasets covering more than 19 years of market history. Our experiments indicate that STANN provides a more effective learning framework than either DNN-based approaches or statistical baselines. We showed that this class of models perform wells in both low- and medium-data settings and that our proposed attention mechanism helps improve forecasting performances over its base model. Finally we illustrated how the use of a forecaster improves autonomous trading strategies.

We would like to emphasize the limited understanding of the relation between our model  effectiveness and the selection of HPs. Indeed, a mis-selection of HPs can have a great impact on a model's performance, potentially hindering its application at a large scale. Hence, we advocate the pursuit of future work to enlarge our theoretical understanding of this class of models, as well as testing to determine whether similar results can be achieved at larger scales and for other TS settings.



§ ACKNOWLEDGEMENTS

This work was supported in part by NSERC CRD - Quebec Prompt - Laplace Insights - EAM - joint program under the grants CRDPJ 537461-18 and 114-IA-Wang-DRC 2019 to Dr. S. Wang, by Mitacs Acceleration under Grant IT13429 to Dr. S. Wang and P. Chatigny and by FQRNT under Grant B2 270188 to P. Chatigny.





















elsarticle-num-names



















