Compact Autoregressive Network

Autoregressive networks can achieve promising performance in many sequence modeling tasks with short-range dependence. However, when handling high-dimensional inputs and outputs, the huge amount of parameters in the network lead to expensive computational cost and low learning efficiency. The problem can be alleviated slightly by introducing one more narrow hidden layer to the network, but the sample size required to achieve a certain training error is still large. To address this challenge, we rearrange the weight matrices of a linear autoregressive network into a tensor form, and then make use of Tucker decomposition to represent low-rank structures. This leads to a novel compact autoregressive network, called Tucker AutoRegressive (TAR) net. Interestingly, the TAR net can be applied to sequences with long-range dependence since the dimension along the sequential order is reduced. Theoretical studies show that the TAR net improves the learning efficiency, and requires much fewer samples for model training. Experiments on synthetic and real-world datasets demonstrate the promising performance of the proposed compact network.


Introduction
Sequence modeling has been used to address a broad range of applications, including macroeconomic time series forecasting, financial asset management, speech recognition and machine translation. Recurrent neural networks (RNN) and their variants, such as Long-Short Term Memory (Hochreiter and Schmidhuber 1997) and Gated Recurrent Unit (Cho et al. 2014), are commonly used as the default architecture or even the synonym of sequence modeling by deep learning practitioners (Goodfellow, Bengio, and Courville 2016). In the meanwhile, especially for high-dimensional time series, we may also consider the autoregressive modeling or multi-task learning, where the output y t and each input y t−i are N -dimensional, and the lag P can be very large for accomodating sequential dependence. Some non-recurrent feed-forward networks with convolutional or other specific architectures have been proposed recently for sequence modeling, and are shown to have state-of-the-art accuracy. For example, some autoregressive networks, such as PixelCNN ( Van den Oord et al. 2016b) and WaveNet (Van den Oord et al. 2016a) for image and audio sequence modeling, are compelling alternatives to the recurrent networks. This paper aims at the autoregressive model (1) with a large number of sequences. This problem can be implemented by a fully connected network with NP inputs and N outputs. The number of weights will be very large when the number of sequences N is large, and it will be much larger if the data have long-range sequential dependence. This will lead to excessive computational burden and low learning efficiency. Recently, Du et al. (2018) showed that the sample complexity in training a convolutional neural network (CNN) is directly related to network complexity, which indicates that compact models are highly desirable when available samples have limited sizes.
To reduce the redundancy of parameters in neural networks, many low-rank based approaches have been investigated. One is to reparametrize the model, and then to modify the network architecture accordingly. Modification of architectures for model compression can be found from the early history of neural networks (Fontaine, Ris, and Boite 1997;Grézl et al. 2007). For example, a bottleneck layer with a smaller number of units can be imposed to constrain the amount of information traversing the network, and to force a compact representation of the original inputs in a multilayer perceptron (MLP) or an autoencoder (Hinton and Salakhutdinov 2006). The bottleneck architecture is equivalent to a fully connected network with a low-rank constraint on the weight matrix in a linear network.
Another approach is to constrain the rank of parameter matrices directly. For instance, Denil et al. (2013) demonstrated significant redundancy in large CNNs, and proposed a low-rank structure of weight matrices to reduce it. If we treat weights in a layer as a multi-dimensional tensor, tensor decomposition methods can then be employed to represent the low-rank structure, and hence compress the network. Among these works, Lebedev et al. (2014) applied the CP decomposition for the 4D kernel of a single convo-lution layer to speed up CNN, and Jaderberg, Vedaldi, and Zisserman (2014) proposed to construct a low-rank basis of filters to exploit cross-channel or filter redundancy. Kim et al. (2016) utilized the Tucker decomposition to compress the whole network by decomposing convolution and fully connected layers. The tensor train format was employed in Novikov et al. (2015) to reduce the parameters in fully connected layers. Several tensor decomposition methods were also applied to compress RNNs (Tjandra, Sakti, and Nakamura 2018;Ye et al. 2018;Pan et al. 2019). In spite of the empirical success of low-rank matrix and tensor approaches, theoretical studies for learning efficiency are still limited.
A fully connected autoregressive network for (1) will have N 2 P weights, and it will reduce to Nr + NP r for an MLP with one hidden layer and r hidden units. The bottleneck architecture still has too many parameters and, more importantly, it does not attempt to explore the possible compact structure along the sequential order. We first simplify the autoregressive network into a touchable framework, by rearranging all weights into a tensor. We further apply Tucker decomposition to introduce a low-dimensional structure and translate it into a compact autoregressive network, called Tucker AutoRegressive (TAR) net. It is a special compact CNN with interpretable architecture. Different from the original autoregressive network, the TAR net is more suitable for sequences with long-range dependence since the dimension along the sequential order is reduced.
There are three main contributions in this paper: 1. We innovatively tensorize weight matrices to create an extra dimension to account for the sequential order and apply tensor decomposition to exploit the low-dimensional structure along all directions. Therefore, the resulting network can handle sequences with long-range dependence.
2. We provide theoretical guidance on the sample complexity of the proposed network. Our problem is more challenging than other supervised learning problems owing to the strong dependency on sequential samples and the multitask learning nature. Moreover, our sample complexity analysis can be extended to other feed-forward networks.
3. The proposed compact autoregressive network can flexibly accommodate nonlinear mappings, and offer physical interpretations by extracting explainable latent features.

Linear Autoregressive Network
This section demonstrates the methodology by considering a linear version of (1), and theoretically studies the sample complexity of the corresponding network.

Preliminaries and Background
Notation We follow the notations in Kolda and Bader (2009) to denote vectors by lowercase boldface letters, e.g. a; matrices by capital boldface letters, e.g. A; tensors of order 3 or higher by Euler script boldface letters, e.g. A. For a generic d th -order tensor A ∈ R p1×···×p d , denote its elements by A(i 1 , i 2 , . . . , i d ) and unfolding of A along the n-mode by A (n) , where the columns of A (n) are the n-mode vectors of A, for n = 1, 2, . . . , d. The vectorization operation is denoted by vec(·). The inner product of two tensors A, B ∈ R p1×···×p d is defined as The Frobenius norm of a tensor A is defined as A F = A, A . The mode-n multiplication × n of a tensor A ∈ R p1×···×p d and a matrix B ∈ R qn×pn is defined as for n = 1, . . . , d, respectively. For a generic symmetric matrix A, λ max (A) and λ min (A) represent its largest and smallest eigenvalues, respectively.
Tucker decomposition The Tucker ranks of A are defined as the matrix ranks of the unfoldings of A along all modes, namely rank i (A) = rank(A (i) ), i = 1, . . . , d. If the Tucker ranks of A are r 1 , . . . , r d , where 1 ≤ r i ≤ p i , there exist a tensor G ∈ R r1×···×r d and matrices U i ∈ R pi×ri , such that which is known as Tucker decomposition (Tucker 1966), . With the Tucker decomposition (2), the n-mode matricization of A can be written as where ⊗ denotes the Kronecker product for matrices.

Linear Autoregressive Network
Consider a linear autoregressive network, where h t = y t is the output, A i s are N × N weight matrices, and b is the bias vector. Let x t = (y t−1 , . . . , y t−P ) be the NP -dimensional inputs. We can rewrite it into a fully connected network, for t = 1, . . . , T , where W = (A 1 , ..., A P ) ∈ R N ×NP is the weight matrix. Note that T denotes the effective sample size, which is the number of samples for training. In other words, the total length of the sequential data is T + P .
To reduce the dimension of W , a common strategy is to constrain the rank of W to be r, which is much smaller than N . The low-rank weight matrix W can be factorized as W = AB, where A is a N × r matrix and B is a r × NP matrix, and the network can be transformed into The matrix factorization reduces the number of parameters in W from N 2 P to Nr + NP r. However, if both N and P are large, the weight matrix B is still of large size. We alternatively rearrange the weight matrices A i s into a 3 rd -order tensor W ∈ R N ×N ×P such that W (1) = W ; see Figure 1 for the illustration. The Tucker decomposition can then be applied to reduce the dimension from three modes simultaneously. If the low-Tucker-rank structure is applied on W with ranks r 1 , r 2 , r 3 , the network becomes by Tucker decomposition further reduces the dimension from the other two modes of low-rank structure in (4), while the low-rankness of W only considers the low-dimensional structure on the 1-mode of W but ignores the possible compact structure on the other two modes.
We train the network based on the squared loss. For simplicity, each sequence is subtracted by its mean, so the bias vector b can be disregarded. The weight matrix or tensor in (3), (4) and (5) can be trained, respectively, by minimizing the following ordinary least squares (OLS), low-rank (LR) and low-Tucker-rank (LTR) objective functions, These three minimizers are called OLS, LR and LTR estimators of weights, respectively. The matrix factorization or tensor Tucker decomposition is not unique. Conventionally, orthogonal constraints can be applied to these components to address the uniqueness issue. However, we do not impose any constraints on the components to simplify the optimization and mainly focus on the whole weight matrix or tensor instead of its decomposition.

Sample Complexity Analysis
The sample complexity of a neural network is defined as the training sample size requirement to obtain a certain training error with a high probability, and is a reasonable measure of learning efficiency. We conduct a sample complexity analysis for the three estimators, W OLS , W LR and W LTR , under the high-dimensional setting by allowing both N and P to grow with the sample size T at arbitrary rates.
We further assume that the sequence {y t } is generated from a linear autoregressive process with additive noises, Denote by W 0 = (A 1 , A 2 , . . . , A P ) the true parameters in (6) and by W 0 the corresponding folded tensor. We assume that W 0 has Tucker ranks r 1 , r 2 and r 3 , and require the following conditions to hold.

Condition 1. All roots of matrix polynomial |I
Condition 2. The errors {e t } is a sequence of independent Gaussian random vectors with mean zero and positive definite covariance matrix Σ e , and e t is independent of the historical observations y t−1 , y t−2 , · · · .
Condition 1 is sufficient and necessary for the strict stationarity of the linear autoregressive process. The Gaussian assumption in Condition 2 is very common in highdimensional time series literature for technical convenience.
Multiple sequence data may exhibit strong temporal and inter-sequence dependence. To analyze how dependence in the data affects the learning efficiency, we follow Basu and Michailidis (2015) to use the spectral measure of dependence below.
where z is any point on the complex plane, and define its extreme eigenvalues as By Condition 1, the extreme eigenvalues are bounded away from zero and infinity, 0 < μ min (A) ≤ μ max (A) < ∞. Based on the spectral measure of dependence, we can derive the non-asymptotic statistical convergence rates for the LR and LTR estimators. Note that C denotes a generic positive constant, which is independent of dimension and sample size, and may represent different values. For any positive number a and b, a b and a b denote that there exists C such that a < Cb and a > Cb, respectively.
Theorem 1. Suppose that Conditions 1-2 are satisfied, and the sample size T r 1 r 2 r 3 + Nr 1 + Nr 2 + P r 3 . With probability at least 1 − exp[−C(r 1 r 2 r 3 + Nr 1 + Nr 2 + P r The proofs of Theorems 1 and 2 are provided in the full version, see Wang et al. (2019). The above two theorems present the non-asymptotic convergence upper bounds for LTR and LR estimators, respectively. Both upper bounds take a general form of M d/T , where M captures the effect from dependence across x t , and d denotes the number of parameters in Tucker decomposition or matrix factorization.
Next, we present a minimax lower bound for low-Tuckerrank tensors. The minimax rate is the convergence rate that cannot be improved and has been widely applied in sample complexity analysis. Denote by T(r 1 , r 2 , r 3 ) the set of tensors with ranks at most (r 1 , r 2 , r 3 ).
By Theorems 1-3, the upper bound of LTR estimator matches the minimax lower bound up to some constant, and the LTR estimator is optimal and more efficient than the LR estimator. For a training error > 0, the sample complexity is T = C(r 1 r 2 r 3 + Nr 1 + Nr 2 + P r 3 )/ 2 for the LTR estimator to have W LTR − W 0 F ≤ ; while the sample complexity is T r(N + NP )/ 2 for the LR estimator. Similarly, the OLS estimator can be shown to have the convergence rate of O( N 2 P/T ), and its sample complexity is CN 2 P/ 2 to achieve a training error .
The sample complexity for the linear autoregressive networks with different structures is proportional to the corresponding model complexity, i.e., sample complexity is O(Md/ε 2 ). Compared with the OLS estimator, the LR and LTR estimators benefit from the compact low-dimensional structure and have smaller sample complexity. Among the three linear autoregressive networks, the LTR network has the most compact structure, and hence the smallest sample complexity.
The sample complexity analysis of the autoregressive networks can be extended to the general feed-forward networks for independent x t by replacing M with the inverse of signal-to-noise ratio, and explains why the low-rank structure can enhance the learning efficiency and reduce the sample complexity.

Tucker Autoregressive Net
This section introduces a compact autoregressive network by formulating the linear autoregressive network with the low-Tucker-rank structure (5), and it has a compact multilayer CNN architecture. We call it the Tucker AutoRegressive (TAR) net for simplicity.

Network Architecture
Rather than directly constraining the matrix rank or Tucker ranks of weights in the zero-hidden-layer network, we can modify the network architecture by adding convolutional layers and fully connected layers to exploit low-rank structure. By some algebra, the framework (5) can be rewritten into where X t = (y t−1 , . . . , y t−P ). A direct translation of the low-Tucker-rank structure leads to a multi-layer convolutional network architecture with two convolutional layers and two fully connected layers; see Figure 2 and Table 1.
To be specific, each column in U 2 is a N × 1 kernel and the first layer outputs r 2 1 × P feature maps. Similarly, U 3 represents the convolution with kernel size 1 × P and r 3 channels. These two convolutional layers work as an encoder to extract the r 2 r 3 -dimensional representation of the N × P input X t for predicting y t . Next, a full connection from r 2 r 3 predictor features to r 1 output features with weights G (1) is followed. Finally, a fully connected layer serves as a decoder to N ouputs with weights U 1 .
The neural network architectures corresponding to the low-rank estimator W LR and ordinary least squares estimator without low-dimensional structure W OLS are the onehidden-layer MLP with a bottleneck layer of size r and the zero-hidden-layer fully connected network, respectively.
The CNN representation in Figure 2 has a compact architecture with r 1 r 2 r 3 + Nr 1 + Nr 2 + P r 3 parameters, which is the same as that of the Tucker decomposition. Compared with the benchmark models, i.e., the one-hidden-layer MLP (MLP-1) and zero-hidden-layer MLP (MLP-0), the introduced low-Tucker-rank structure increases the depth of the network while reduces the total number of weights. When Tucker ranks are small, the total number of parameters in our network is much smaller than those of the benchmark networks, which are r(N + NP ) and N 2 P , respectively.
To capture the complicated and non-linear functional mapping between the prior inputs and future responses, non-linear activation functions, such as rectified linear unit (ReLU) or sigmoid function, can be added to each layer in the compact autoregressive network. Hence, the additional depth from transforming a low-Tucker-rank single layer to a multi-layer convolutional structure enables the network to approximate the target function better. The linear network without activation in the previous section can be called the linear TAR net (LTAR).

Separable Convolutional Kernels
Separable convolutions have been extensively studied to replace or approximate large convolutional kernels by a series of smaller kernels. For example, this idea was explored in multiple iterations of the Inception blocks (Szegedy et al.

Feedforward
Layer Content and explanation Dimensions No. of parameters Z 2 := U 3 Z 1 1 × P convolutions r 2 r 3 feature maps 1 × 1 P r 3 F1 Z 3 := G (1) vec(Z 2 ) full connection response factor loadings r 1 × 1 r 1 r 2 r 3 OUTPUT Z 4 := h t = U 1 Z 3 full connection output prediction N × 1 Nr 1  2015; 2016; 2017) to decompose a convolutional layer with a 7 × 7 kernel into that with 1 × 7 and 7 × 1 kernels. Tensor decomposition is an effective method to obtain separable kernels. In our TAR net, these two convolutional layers extract the information from inputs along the columnwise direction and row-wise direction separately. Compared with the low-rank matrix structure, the additional decomposition in the Tucker decomposition along the second and third modes segregates the full-sized convolutional kernel into r 2 r 3 pairs of separable kernels.

Two-Lane Network
If no activation function is added, the first two row-wise and column-wise convolutional layers are exchangeable. However, exchanging these two layers with nonlinear activation functions can result in different nonlinear approximation and physical interpretation.
For the general case where we have no clear preference on the order of these two layers, we consider a two-lane network variant, called TAR-2 network, by introducing both structures into our model in parallel followed by an average pooling to enhance the flexibility; see Figure 3 in  for a graphical illustration.

Implementation
Details We implement our framework on PyTorch, and the Mean Squared Error (MSE) is the target loss function. The gradient descent method is employed for the optimization with learning rate and momentum being 0.01 and 0.9, respectively. If the loss function drops by less than 10 −8 , the procedure is then deemed to have reached convergence.
Hyperparameter tuning In the TAR net, the sequential dependence range P and the Tucker ranks r i , are prespecified hyperparameters. Since cross-validation cannot be applied to sequence modeling, we suggest tuning hyperparameters by grid search and rolling forecasting performance.

Experiments
This section first performs analysis on two synthetic datasets to verify the sample complexity established in Theorems 1-3 and to demonstrate the capability of TAR nets in nonlinear functional approximation. Three real datasets are then analyzed by the TAR-2 and TAR nets, together with their linear counterparts. For the sake of comparison, some benchmark networks, including MLP-0, MLP-1, Recurrent Neural Network (RNN) and Long Short-Term Memory (LSTM), are also applied to the dataset.

Numerical Analysis for Sample Complexity
Settings In TAR net or the low-Tucker-rank framework (5), the hyperparameters, r 1 , r 2 and r 3 , are of significantly smaller magnitude than N or P , and are equally set to 2 or 3. As sample complexity is of prime interest rather than the range of sequential dependence, we let P equal to 3, 5 or 8. For each combination of (r 1 , r 2 , r 3 , P ), we consider N = 9, 25 and 36, and the sample size T is chosen such that N/T = (0.15, 0.25, 0.35, 0.45).

Data generation
We first generate a core tensor G ∈ R r1×r2×r3 with entries being independent standard normal random variables, and then rescale it such that the largest singular value of G (1) is 0.9. For each 1 ≤ i ≤ 3, the leading r i singular vectors of random standard Gaussian matrices are used to form U i . The weight tensor W 0 can thereby be reconstructed, and it is further rescaled to satisfy Condition 1. We generate 200 sequences with identical W 0 . The first 500 simulated data points at each sequence are discarded to alleviate the influence of the initial values. We apply the MLP-0, MLP-1 and LTAR to the synthetic dataset. The averaged estimation errors for the corresponding OLS, LR, and LTR estimators are presented in Figure 3.
Results The x-axis in Figure 3 represents the ratio of N/T , and the y-axis represents the averaged estimation error in Frobenius norm. Along each line, as N is set to be fixed, we obtain different points by readjusting the sample size T . Roughly speaking, regardless of the models and parameter settings, estimation error increases with varying rates as the sample size decreases. The rates for OLS rapidly become explosive, followed by LR, whereas LTR remains approximately linear, which is consistent with our findings at Theorems 1 and 2.
Further observation reveals that the increase in P predominantly accelerates the rates for OLS and LR, but appears to have an insignificant influence on the estimation error from LTR.
For the case with P = 8, instability of the estimation error manifests itself in LR under insufficient sample size, say when N/T is as large as 0.35. This further provides the rationale for dimension reduction along sequential order. When N/T = 0.45, the solution is not unique for both OLS and LR, and consequently, these points are not shown in the figure.

Numerical Analysis for Nonlinear Approximation
Settings The target of this experiment is to compare the expressiveness of LTAR, TAR and TAR-2 nets. The conjecture is that, regardless of the data generating process, TAR-2 and TAR nets under the same hyperparameter settings as the LTAR net would have an elevated ability to capture nonlinear features. We set (r 1 , r 2 , r 3 , N, P ) = (2, 2, 2, 25, 3), and have also tried several other combinations. Similar findings can be observed, and the results are hence omitted here.
Data generation Two data generating processes are considered to create sequences with either strictly linear or highly nonlinear features in the embedding feature space. We refer to them as L-DGP and NL-DGP, respectively. L-DGP is achieved by randomly assigning weights to LTAR layers and producing a recursive sequence with a given initial input matrix. NL-DGP is attained through imposing a nonlinear functional transformation to the low-rank hidden layer of an MLP. In detail, we first transformed a N × P matrix to a r 1 × r 2 low-rank encoder. Then, we applied a nonlinear mapping f (·) = cos(1/ · F ) to the encoder, before going through a fully connected layer to retrieve an output of size N × 1.

Implementation & Evaluation
In this experiment, we use L-DGP and NL-DGP to separately generate 200 data sequences, which are fitted by TAR-2, TAR and LTAR nets. The sequence lengths are chosen to be either 101 or 501. For each sequence, the last data point is retained as a single test point, whereas the rest are used in model training. We adopt three evaluation metrics, namely, the averaged L 2 norm between prediction and true value, the standard Root-Mean-Square Error (RMSE), and Mean Absolute Error (MAE). The results are given in Table 2.
Results When the data generating process is linear (L-DGP), the LTAR net reasonably excels in comparison to the other two, obtaining the smallest L 2 -norm, RMSE and MAP. TAR-2 yields poorer results for a small sample size of 100 due to possible overparametrization. However, its elevated expressiveness leads it to outperform TAR when T = 500.
For the nonlinear data generating process (NL-DGP), as we expect, the TAR-2 and TAR nets with nonlinear structure outperform the LTAR net. In the meanwhile, as the exchangeability of latent features holds, the TAR-2 net seems to suffer from model redundancy and thereby performs worse than the TAR net.

Real Datasets
Dataset We use the three publicly available datasets. 1. USME dataset: It contains 40 US macroeconomic variables provided in Koop (2013), including consumption, production indices, stock market indicators and the interest rates. The data series are taken quarterly from 1959 to 2007 with a total of 194 observed time points. 2. GHG dataset: We retrieve a partial greenhouse gas (GHG) concentration dataset (Lucas et al. 2015) from UCI repository containing the time series of GHG tracers. The observations are gathered from 45 sensors, averaged to be spaced 12 hours apart with 327 time points. . We set (r 1 , r 2 , r 3 ) = (2, 2, 2) for subplots in the left column and (3, 3, 3) for subplots in the right column. And upper, middle and lower panels refer to cases with P = 3, 5 and 9.
3. Traffic dataset: The data record the hourly road occupancy rate (between 0 and 1) in the San Francisco Bay freeway (Lai et al. 2018) obtained by 30 different sensors in 2015.
In the preprocessing step, the series were transformed to be stationary before further standardization; details see the full version .
Models for comparison For the sake of comparison, besides the proposed models, TAR-2, TAR and LTAR, we also consider four other commonly used networks in the literature with well-tuned hyperparameters. The first two are the previously mentioned MLP-0 and MLP-1. The remaining two are RNN and LSTM, which are two traditional sequence modeling frameworks. RNN implies an autoregressive moving average framework and can transmit extra useful information through the hidden layers. It is hence expected to outperform an autoregressive network. LSTM is advantageous in many sequence learning tasks, but may be more susceptible to small sample size. As a result, RNN and LSTM with the optimal tuning hyperparameters serve as our benchmarks.  Implementation We tune the parameters (P, r 1 , r 2 , r 3 ) for our proposed models. To ensure a fair comparison, the size of the hidden layer in MLP-1 is set to be P , and we consider only one hidden layer for RNN and LSTM. The number of neurons in the hidden layer is treated as a tunable hyperparameter. The bias terms are added back to the TAR-2, TAR and LTAR nets for expansion of the model space.
For each network, one-step-ahead forecasting is carried out recursively. In other words, the trained network predicts one future step, and immediately includes the new observation in the training set for the prediction of the next step. The averaged L 2 -norm, RMSE and MAP are used as the evaluation criteria.
For the USME dataset, the parameters are chosen as (P, r 1 , r 2 , r 3 ) = (4, 4, 3, 2). The first 104 time points of each series are used as the first training samples with an effective sample size of 100, whereas the rolling forecast procedure is applied to the rest 90 test samples. For GHG dataset, the parameters are set as (P, r 1 , r 2 , r 3 ) = (20, 2, 4, 5) with an effective training sample size of 100 and testing sample size of 207. For the Traffic dataset, the parameters are set as (P, r 1 , r 2 , r 3 ) = (30, 2, 6, 8) and we choose a larger training size of 200 with 70 testing samples.
T-test for significance We adopt a paired two-sample ttest for the significant difference of L 2 -norm between TAR-2 and RNN/LSTM, separately. The hypotheses are H 0 : μ L2 (TAR-2) = μ L2 (RNN/LSTM) against μ L2 (TAR-2) > μ L2 (RNN/LSTM). Note that each forecast error is a vector of R N , and there are n steps of prediction. Consider the L 2norm for each forecast error, and it then leads to a sequence of n scalars, which can be assumed to be independent; see Kuester, Mittnik, and Paolella (2006).
For the USME data, the p-value of the test between TAR-2 and LSTM is 0.031, and between TAR-2 and RNN is 0.203. For the GHG data, the tests for comparing TAR-2 to both RNN and LSTM are significant with p-values smaller than 0.01. For the Traffic data, the test between TAR-2 and LSTM has a p-value close to 0. While for the comparison of TAR-2 and RNN, the p-value is 0.052.  Results From Table 3, the proposed TAR-2 and TAR nets rank top two in terms of one-step-ahead rolling forecast performance, exceeding the fine-tuned RNN model with the size of the hidden layer equal to one. The two-lane network TAR-2 outperforms the one-lane network TAR emphasizing its ability to capture non-exchangeable latent features. According to our experiments, the performance of both RNN and LSTM deteriorates as the dimension of the hidden layer increases, which indicates that overfitting is a serious issue for these two predominate sequence modeling techniques.

Conclusion and Discussion
This paper rearranges the weights of an autoregressive network into a tensor, and then makes use of the Tucker decomposition to introduce a low-dimensional structure. A compact autoregressive network is hence proposed to handle the sequences with long-range dependence. Its sample complexity is also studied theoretically. The proposed network can achieve better prediction performance on a macroeconomic dataset than some state-of-the-art methods including RNN and LSTM. For future research, this work can be improved in three directions. First, our sample complexity analysis is limited to linear models, and it is desirable to extend the analysis to networks with nonlinear activation functions. Secondly, the dilated convolution in WaveNet (Van den Oord et al. 2016a), can reduce the convolutional kernel size, and hence can efficiently access the long-range historical inputs. This structure can be incorporated into our framework to further compress the network. Finally, a deeper autoregressive network can be constructed to enhance the expressiveness of nonlinearity.