\section{Introduction}

Ensemble methods are considered to be a cornerstone of modern machine learning \cite{Ree18}. They are commonly used for regression and classification problems.
Ensembling is also a very effective way of increasing the predictive power of forecasting models. Combining many base models improves the final forecasting accuracy as well as the stability of the response when compared to a single model approach.
Success in ensemble learning depends on the proper flexibility of the ensemble members and the trade-off between their performance and diversity \cite{Bro05}. It is also determined by the way learners are generated at the successive stages of ensembling and the method employed to combine them.

The effectiveness of ensembling in forecasting is evidenced by the fact that in the most renowned forecasting competition, M4 \cite{Mar18}, of the 17 most accurate models, 12 used ensembling in some form \cite{Ati20}. The winning submission, which is a hybrid model combining exponential smoothing and long short-term memory, used three types of ensembling simultaneously \cite{Smyl20}: combining results of the stochastic training process, bagging, and combining multiple runs.

To improve the performance of ensemble learning many approaches have been proposed such as stacking \cite{Wol92}, bagging \cite{Bre96}, boosting \cite{Dru99}, negative correlation learning \cite{Chen09}, snapshot ensembles \cite{Hua17},
and horizontal and vertical ensembles developed for deep learning \cite{Xie13}. Boosting, which this work focuses on, is a general ensemble technique that involves sequentially adding base models to the ensemble where subsequent models correct the performance of prior models. This approach is very effective
as evidenced by the high ranking positions of boosted models such as XGBoost \cite{Chen16}, i.e. ensemble of decision trees with regularized gradient boosting, in competitions such as those organized by Kaggle. Boosting also has many applications in the forecasting field. Some examples are: \cite{Zie16}, where an ensemble of boosted trees is used for bankruptcy prediction;
\cite{Ni20}, where XGBoost is combined with a Gaussian mixture model for monthly streamflow forecasting;
\cite{Li19}, where AdaBoost is applied as a component of a hybrid model for multi-step wind speed forecasting;
and \cite{Mit22}, where a natural gradient boosting algorithm is applied for solar power probabilistic forecasting.
This last work highlights a valuable advantage of ensembling. It can produce probabilistic forecasts, i.e. the distribution of the forecasted variable in the future.

In this study, we propose a boosted version of our ensemble of randomized neural networks (RandNNs) for complex time series forecasting \cite{Dud21}. In \cite{Dud21}, we focused on strategies for controlling the diversity of ensemble members. The members were trained independently. Here, we construct an ensemble sequentially. When a new member is added to the ensemble, it learns by taking into account the results of the ensemble members so far. We consider three methods of boosting. The contribution of this study is threefold:

\begin{enumerate}
    \item
    We propose new methods of ensemble boosting: ensemble learning based on corrected targets and ensemble learning based on opposed response. They are both employed to ensure similar tasks for all ensemble members. This unification of the tasks justifies the use of identical base models in terms of architecture and hyperparameters at all stages of ensembling.

    \item
    We develop three ensemble learning approaches for forecasting complex time series with multiple seasonality. They are based on RandNNs, pattern-based time series representation and three methods of boosting: based on residuals, corrected targets and opposed response.

    \item
    We empirically compare the performance of the proposed boosted ensemble methods on challenging short-term load forecasting problems with triple seasonality, and conclude that the opposed response-based approach outperforms its competitors in terms of forecasting accuracy and sensitivity to hyperparameters.

\end{enumerate}

The rest of the work is organized as follows. In Section II, we present a base model, RandNN. Details of the proposed three methods of boosted ensemble learning are described in Section III. The experimental framework used to evaluate and compare the proposed ensemble methods is described in Section IV. Finally, Section V concludes the work.

\section{Base Forecasting Model - RandNN}

As a base model, we use a single hidden layer feedforward NN with $m$ logistic sigmoid hidden nodes \cite{Dud21a}. In randomized learning, the weights of hidden nodes are selected randomly from a uniform distribution and symmetrical interval $U=[-u,u]$. The biases of these nodes are calculated, according to recent research \cite{Dud19}, based on the weights as follows: $b_j = -\mathbf{a}_j^T\mathbf{x}^*_j$, where $\mathbf{a}_j$ is the vector of weights for the $j$-th hidden node, and $\mathbf{x}^*_j$ is one of the training patterns selected at random (see \cite{Dud19} and \cite{Dud20} for details, justification and other variants). This way of generating hidden nodes places the sigmoids into the input feature space limited to some hypercube, avoiding their saturated parts. Moreover, the sigmoids are distributed according to the data distribution. These improve the aproximation and generalization abilities of the model \cite{Dud19}, \cite{Dud20}.

The hidden node sigmoids are combined linearly by the output nodes: $\varphi_k(\mathbf{x})=\sum_{j=1}^{m}\beta_{j,k}h_j(\mathbf{x})$, where $h_j(\mathbf{x})$ is the output of the $j$-th hidden node, and $\beta_{j,k}$ is the weight between $j$-th hidden and $k$-th output nodes. The only learnable parameters are the output weights. They are calculated from $\boldsymbol{\beta} = \mathbf{H}^+\mathbf{Y}$, where $\mathbf{H}$ is a matrix of the hidden layer outputs, $\mathbf{H}^+$ denotes its Moore–Penrose generalized inversion, and $\mathbf{Y}$ is a~matrix of target output patterns. $\mathbf{H}$ is a nonlinear feature mapping from $n$-dimensional input space to $m$-dimensional projection space (usually $m \gg n$). Note that this projection is random. Due to the fixed parameters of the hidden nodes, the optimization problem in randomized learning (selection of weights $\boldsymbol{\beta}$) becomes convex and can be easily solved by the standard least-squares method.

The forecasting model based on RandNN, which we adapt from \cite{Dud21a}, has two more components: encoder and decoder.
Let us consider time series $\{E_k\}_{k=1}^K$ with multiple seasonality. The encoder transforms this series into input and output patterns expressing seasonal sequences of the shortest length. The input patterns, $\mathbf{x}_i = [x_{i,1}, …, x_{i,n}]^T$, represent sequences $ \mathbf{e}_i = [E_{i,1},…, E_{i,n}]^T $, while the output patterns, $\mathbf{y}_i = [y_{i,1},…, y_{i,n}]^T$, represent forecasted sequences $ \mathbf{e}_{i+\tau} = [E_{i+\tau,1},…, E_{i+\tau,n}]^T $, where $n$ is a period of the seasonal cycle (e.g. 24 hours for daily seasonality), $i=1, ..., K/n$ is the sequence number, and $\tau \geq 1$ is a forecast horizon. The patterns are defined as follows:

\begin{equation}\label{eq1}
\mathbf{x}_i = \frac{\mathbf{e}_i-\overline{e}_i}{\widetilde{e}_i},
\qquad
\mathbf{y}_i = \frac{\mathbf{e}_{i+\tau}-\overline{{e}}_i}{\widetilde{{e}}_i}
\end{equation}
where $\overline{{e}}_i$ is the mean value of sequence $\mathbf{e}_i$, and $\widetilde{e}_i = \sqrt{\sum_{t=1}^{n} (E_{i,t}-\overline{e}_i)^2}$ is a measure of sequence $\mathbf{e}_i$ dispersion.

Input patterns $\mathbf{x}_i$ represent successive seasonal sequences which are centered and normalized. They have a zero mean, and the same variance and unity length. Thus they are unified and differ only in shape. In contrast, patterns $\mathbf{y}_i$ are not globally unified and can express additional seasonality, e.g. when time series include both daily and weekly seasonalities, the former is expressed in the y-pattern shape, while the latter is expressed in y-pattern level and dispersion (compare x- and y-patterns in Fig. 2 and see discussion in \cite{Dud21a}). In the case of such seasonalities, we build forecasting models that learn from data representing the same days of the week, e.g. for test query pattern $\mathbf{x}$ representing Monday, training set $\Phi$ consists of x-patterns representing all historical Mondays from the data and y-pattern representing the Tuesdays following them  (assuming $\tau = 1$).

The decoder based on the y-pattern forecasted by the network, $\widehat{\mathbf{y}}$, and coding variables describing the query sequence, $\widetilde{{e}}$ and $\overline{{e}}$, using transformed equation \eqref{eq1} for $\mathbf{y}$, calculates the forecasted seasonal sequence:

\begin{equation}\label{eq3}
\widehat{\mathbf{e}} = \widehat{\mathbf{y}}\widetilde{{e}}+\overline{{e}}
\end{equation}

Remarks:

\begin{enumerate}
    \item
RandNN was designed for forecasting time series with multiple seasonalities. In the case of one seasonality, y-patterns express only one seasonality, and there is no need to decompose the forecasting problem. In the case of no seasonality, the input pattern length should be selected experimentally, while the y-pattern length is equal to the forecast horizon.

    \item
The bounds of the interval for random weights, $u$, correspond to the maximum sigmoid slope. To increase interpretability, let us express the bounds $u$ using the slope angles $\alpha_{max}$ \cite{Dud20}: $u=4 \tan \alpha_{max}$, and treat $\alpha_{max}$ as a hyperparameter. The second hyperparameter is the number of hidden nodes, $m$. Both hyperparameters decide about the bias-variance tradeoff of the model and should be tuned to the complexity of the target function.

    \item
Advantages of RandNN are extremly fast training and the simplification of the forecasting task due to pattern representation.
In \cite{Dud21a}, it was shown that RandNN can compete with fully-trained NN in terms of forecasting accuracy, but is much faster to train.

\end{enumerate}

\section{Boosting of Ensemble Learning}

\subsection{Ensemble Learning Based on Residuals}

An ensemble based on residuals, which is a simplified variant of a gradient boosting algorithm \cite{Mas99}, is constructed sequentially. In the first step, a base model learns on the training set $\{(x_i,y_i)\}_{i=1}^N$ (we consider a scalar input and output for simplicity) and fits to original data. Let us denote this model $f_1(x)$ and the ensemble model including one member $F_1(x)=f_1(x)$. In the second step, the second base model is added, $f_2(x)$, such that the sum of this model with the previous one, $F_2(x)=F_1(x)+f_2(x)$, is the closest possible to the target $y$. In the successive steps, further base models are added with the same expectation. In the $k$-th step, the function fitted by the ensemble is $F_k(x)=F_{k-1}(x)+f_k(x)$. Thus, the error in this step, $MSE_k=\frac{1}{N}\sum_{i=1}^{N} \left(y_i-F_k(x_i)\right)^2$, can be written as:

\begin{equation}
MSE_k=\frac{1}{N}\sum_{i=1}^{N}\left(f_k(x_i)-r_{k-1,i}\right)^2
\label{MSE}
\end{equation}
where $r_{k-1,i} = y_i-F_{k-1}(x_i)$ is a residual between the target and the response of the ensemble of $k-1$ members.

Equation \eqref{MSE} clearly shows that the base model added to the ensemble at the $k$-th stage fits to the residuals between the target and the ensemble built at stage $k-1$. So, it attempts to correct the errors of its predecessors. The training set for the base model in the $k$-th stage is $\{(x_i,r_{k-1,i})\}_{i=1}^N$. The final ensemble response is the sum of all member responses: $F_K(x)=\sum_{k=1}^{K} f_k(x)$.

\begin{algorithm}[h]
	\caption{EnsR: Boosting based on residuals}
	\label{alg1}
	\begin{algorithmic}

		\STATE {\bfseries Input:} Base model $f$ (RandNN), Training set $\Phi = \{(\mathbf{x}_i, \mathbf{y}_i)\}_{i=1}^N$, Ensemble size $K$\\

		\STATE {\bfseries Output:} RandNN ensemble $F_K$\\
		\STATE {\bfseries Procedure:}\\
		\FOR{$k=1$ {\bfseries to} $K$}
		\STATE Learn $f_k$ based on $\Phi$ \\
		\STATE Calculate ensemble response $F_k(\mathbf{x}_i)=\sum_{l=1}^{k} f_l(\mathbf{x}_i), i = 1, ..., N$\\
        \STATE Determine residuals $\mathbf{r}_{k,i}=\mathbf{y}_i-F_k(\mathbf{x}_i), i = 1, ..., N$\\
        \STATE Modify training set $\Phi = \{(\mathbf{x}_i, \mathbf{r}_{k,i})\}_{i=1}^N$ \\
		\ENDFOR
	\end{algorithmic}
\end{algorithm}

Algorithm \ref{alg1} and Fig. \ref{FigEnsR} demonstrate the process of building an ensemble based on residuals (EnsR) for pattern-based forecasting. Learner 1 (RandNN) learns the original target function on $\Phi$. Each subsequent learner learns the residuals between the target patterns $\textbf{y}$ and the aggregated outputs of its predecessors shown in the lower panel of Fig. \ref{FigEnsR}. Note that in EnsR, each learner can have a different problem to solve, expressing different features. Learner 1 learns the specific patterns of seasonal cycles $\textbf{y}$, while the next learners learn completely different tasks, i.e. the residuals, which do not have such distinct patterns as $\textbf{y}$ and have a large stochastic component. This inconsistency between the problems solved by the learners can affect negatively the final result, especially in the common case of using identical base models (the same architecture and hyperparameters) at every stage of ensembling.

\begin{figure}[h]
	\centering
	\includegraphics[width=0.22\textwidth]{Res1.eps}
    \includegraphics[width=0.22\textwidth]{Res2.eps}
    \includegraphics[width=0.25\textwidth]{Res4.eps}
    \includegraphics[width=0.22\textwidth]{Res6.eps}
    \includegraphics[width=0.45\textwidth]{Res3.eps}
    \includegraphics[width=0.25\textwidth]{Res5.eps}
    \includegraphics[width=0.22\textwidth]{Res7.eps}

    \caption{EnsR: Responses produced by individual learners (upper panel) and the ensemble (lower panel). Target responses of individual learners in blue, target responses of the ensemble in black, real responses in red.}
    \label{FigEnsR}
\end{figure}

\subsection{Ensemble Learning Based on Corrected Targets}

To unify the problems solved by the learners at successive stages of ensembling, we modify the EnsR framework as follows. During the sequential process, at stage $k$, the base model $f_k(x)$ is added to the ensemble. Ensemble response at this stage is the average of $k$ learners: $F_k(x)=\frac{1}{k}\sum_{l=1}^{k} f_l(x)$. The loss function can be expressed as:

\begin{equation}
\begin{split}
MSE_k & =\frac{1}{N}\sum_{i=1}^{N}\left(y- \left(\frac{1}{k}\sum_{l=1}^{k-1} f_l(x) + \frac{f_k(x)}{k}\right)\right)^2\\
& =\frac{1}{Nk}\sum_{i=1}^{N}\left(f_k(x)-\left(ky-\sum_{l=1}^{k-1} f_l(x)\right)\right)^2
\end{split}
\label{MSE2}
\end{equation}

As can be seen from this equation, the difference between $ky$ and the sum of the previous learners is the new target to which the $k$-th learner is fitted. This target expresses pattern $y$ corrected by aggregated residuals of the previous learners: $y+\sum_{l=1}^{k-1} (y-f_l(x))$. If the aggregated residuals are much smaller compared to the y-pattern (we expect this), the new targets at all stages of ensembling have a similar shape to the y-pattern. So, all learners have similar tasks to solve and it is justified for them to have the same architecture and hyperparameters. This unburdens us from the awkward and time-consuming task of selecting the optimal model at each stage of ensembling.

Algorithm \ref{alg2} summarizes ensemble learning based on corrected targets (EnsCT) for pattern-based forecasting. Fig. \ref{FigEnsR2} shows the targets for learners at successive stages, learners' outputs and the ensemble outputs. It was observed that at the initial stages of ensembling, the targets express the y-pattern shape, but this shape degrades gradually in the later stages - see the target for $K$-th learner in Fig. \ref{FigEnsR2}. Thus, the proposed EnsCT only partially solves the problem of inconsistency between tasks learned at the successive stages of ensembling.

\begin{algorithm}[h]
	\caption{EnsCT: Boosting based on corrected targets}
	\label{alg2}
	\begin{algorithmic}

		\STATE {\bfseries Input:} Base model $f$ (RandNN), Training set $\Phi = \{(\mathbf{x}_i, \mathbf{y}_i)\}_{i=1}^N$, Ensemble size $K$\\

		\STATE {\bfseries Output:} RandNN ensemble $F_K$\\
		\STATE {\bfseries Procedure:}\\
		\FOR{$k=1$ {\bfseries to} $K$}
		\STATE Learn $f_k$ based on $\Phi$ \\
		\STATE Calculate ensemble response $F_k(\mathbf{x}_i)=\frac{1}{k}\sum_{l=1}^{k} f_l(\mathbf{x}_i), i = 1, ..., N$\\
        \STATE Determine corrected targets $\mathbf{y}'_{k,i}=(k+1)\mathbf{y}_i-kF_k(\mathbf{x}_i), i = 1, ..., N$\\
        \STATE Modify training set $\Phi = \{(\mathbf{x}_i, \mathbf{y}'_{k,i})\}_{i=1}^N$ \\
		\ENDFOR
	\end{algorithmic}
\end{algorithm}

\begin{figure}[h]
	\centering
	\includegraphics[width=0.22\textwidth]{ResC1.eps}
    \includegraphics[width=0.22\textwidth]{ResC2.eps}
    \includegraphics[width=0.25\textwidth]{ResC4.eps}
    \includegraphics[width=0.22\textwidth]{ResC6.eps}
    \includegraphics[width=0.45\textwidth]{ResC3.eps}
    \includegraphics[width=0.25\textwidth]{ResC5.eps}
    \includegraphics[width=0.22\textwidth]{ResC7.eps}

    \caption{EnsCT: Responses produced by individual learners (upper panel) and the ensemble (lower panel). Target responses of individual learners in blue, target responses of the ensemble in black, real responses in red.}
    \label{FigEnsR2}
\end{figure}

\subsection{Ensemble Learning Based on Opposed Response}

To prevent the degradation of the target shapes in the subsequent steps of ensembling, we propose ensemble learning based on opposed response (EnsOR).
The first step of the ensemble building procedure is the same as in EnsR and EnsCT.
At this stage, the base model $f_1(x)$ learns on original training set $\Phi$. The ensemble response is $F_1(x)=f_1(x)$.
The residual is calculated, $r_1 = y-F_1(x)$, and the "opposed" response pattern is determined as follows:

\begin{equation}
\hat{y}'_1=y+r_1=2y-F_1(x)
\label{Opp}
\end{equation}

The opposed response pattern expresses the target pattern augmented by the opposed error produced by the ensemble (see Fig. \ref{figOP}). The average of the response pattern $F_1(x)$ and the opposed response pattern $\hat{y}'_1$ gives the target pattern $y$. In the next step, base model $f_2(x)$ learns on training set $\{(x_i,\hat{y}'_{1,i})\}_{i=1}^N$. Thus it learns the opposed response to reduce the ensemble residual. The ensemble response is calculated as the average of learners: $F_2(x)=\frac{f_1(x)+f_2(x)}{2}$. These operations are repeated in the following steps (see Algorithm \ref{alg3}). Namely, in step $k$, the opposed response pattern determined at stage $k-1$, $\hat{y}'_{k-1}=2y-F_{k-1}(x)$, becomes the target pattern for learner $f_k(x)$. The ensemble response is the average of $k$ learners and the loss function at stage $k$ takes the form:

\begin{equation}
MSE_k=\frac{1}{N}\sum_{i=1}^{N}\left(f_k(x_i)-\hat{y}'_{k-1,i}\right)^2
\label{MSE2}
\end{equation}
where $\hat{y}'_{k-1,i}=y_i+r_{k-1,i}=2y_i-F_{k-1}(x_i)$ is the opposed response pattern at stage $k-1$.

\begin{figure}
	\centering
	\includegraphics[width=0.5\textwidth]{Cor0.eps}
    \caption{Construction of the opposed response pattern.}
    \label{figOP}
\end{figure}

\begin{algorithm}[h]
	\caption{EnsOR: Boosting based on opposed response}
	\label{alg3}
	\begin{algorithmic}

		\STATE {\bfseries Input:} Base model $f$ (RandNN), Training set $\Phi = \{(\mathbf{x}_i, \mathbf{y}_i)\}_{i=1}^N$, Ensemble size $K$\\

		\STATE {\bfseries Output:} RandNN ensemble $F_K$\\
		\STATE {\bfseries Procedure:}\\
		\FOR{$k=1$ {\bfseries to} $K$}
		\STATE Learn $f_k$ based on $\Phi$ \\
		\STATE Calculate ensemble response $F_k(\mathbf{x}_i)=\frac{1}{k}\sum_{l=1}^{k} f_l(\mathbf{x}_i), i = 1, ..., N$\\
        \STATE Determine opposed response $\hat{\mathbf{y}}'_{k,i}=2\mathbf{y}_i-F_k(\mathbf{x}_i), i = 1, ..., N$\\
        \STATE Modify training set $\Phi = \{(\mathbf{x}_i, \hat{\mathbf{y}}'_{k,i})\}_{i=1}^N$ \\
		\ENDFOR
	\end{algorithmic}
\end{algorithm}

Fig. \ref{figEnsOR} shows learners' and ensemble responses in the following steps of ensembling. Dashed lines express the opposed responses which become targets for the based models in the next steps (blue lines). Note that the shape of the initial target pattern $\mathbf{y}$ is maintained until the last stage. Thus, all learners learn on similar data, and using identical base models at each stage of ensembling means no objections can be raised, unlike in the case of EnsR.

\begin{figure}[h]
	\centering
	\includegraphics[width=0.22\textwidth]{Cor1.eps}
    \includegraphics[width=0.22\textwidth]{Cor2.eps}
    \includegraphics[width=0.25\textwidth]{Cor4.eps}
    \includegraphics[width=0.22\textwidth]{Cor6.eps}
    \includegraphics[width=0.45\textwidth]{Cor3.eps}
    \includegraphics[width=0.25\textwidth]{Cor5.eps}
    \includegraphics[width=0.22\textwidth]{Cor7.eps}

    \caption{EnsOR: Responses produced by individual learners (upper panel) and the ensemble (lower panel). Target responses of individual learners in blue, target responses of the ensemble in black, real responses in red, and opposed responses in dashed red.}
    \label{figEnsOR}
\end{figure}

\section{Experimental Study}

In this section, we verify our proposed methods of ensemble boosting on four time series forecasting problems. These comprise short-term electrical load forecasting problems for four European countries: Poland (PL), Great Britain (GB), France (FR) and Germany (DE) (data was collected from \url{www.entsoe.eu}). The hourly load time series express three seasonalities: yearly, weekly and daily. The data period is four years, from 2012 to 2015. Atypical days such as public holidays were excluded from the data (between 10 and 20 days a year). The forecasting problem is to predict the load profile (24 hourly values) for each day of 2015 based on historical data. The forecast horizon is one day, $\tau=1$. The number of ensemble members was $K=50$. As a performance metric we use mean absolute percentage error (MAPE).

In the first experiment, we evaluate ensemble sensitivity to the base model hyperparameters: number of hidden nodes $m$ and size of the interval for hidden weights $\alpha_{max}$. Fig. \ref{figSens} shows the impact of hyperparameters on the test MAPE. Note that EnsR is the most sensitive to hyperparameters, while EnsOR is the least sensitive.
To quantitatively compare the sensitivities of the ensemble variants, as a rough measure of sensitivity to a given hyperparameter, we define standard deviation of the test MAPE for the ensemble with optimal values of other hyperparameters:

\begin{equation}
S_m = Std(MAPE(\mathbf{y},F(\mathbf{x},\alpha^*_{max},m)))
\label{Sm}
\end{equation}
\begin{equation}
S_{\alpha_{max}} = Std(MAPE(\mathbf{y},F(\mathbf{x},\alpha_{max},m^*)))
\label{Sa}
\end{equation}
where the optimal hyperparameter values are marked with asterisks.

\begin{figure}[h]
	\centering
	\includegraphics[width=0.7\textwidth]{Sens1a.eps}
	\includegraphics[width=0.042\textwidth]{legend1.eps}
    \includegraphics[width=0.7\textwidth]{Sens2a.eps}
    \includegraphics[width=0.042\textwidth]{empty.eps}
    \includegraphics[width=0.7\textwidth]{Sens3a.eps}
    \includegraphics[width=0.042\textwidth]{empty.eps}
    \includegraphics[width=0.7\textwidth]{Sens4a.eps}
    \includegraphics[width=0.042\textwidth]{empty.eps}
    \caption{Ensemble sensitivity to RandNN hyperparameters.}
    \label{figSens}
\end{figure}

\begin{figure}[h]
	\centering
	\includegraphics[width=0.24\textwidth]{PLk.eps}
    \includegraphics[width=0.24\textwidth]{GBk.eps}
    \includegraphics[width=0.24\textwidth]{FRk.eps}
    \includegraphics[width=0.24\textwidth]{DEk.eps}
    \caption{Ensemble error at successive stages.}
    \label{figk}
\end{figure}

From Fig. \ref{figSens}, we can see that the accuracy of EnsR and EnsCT deteriorates quickly with the number of hidden nodes and interval $U$ size, when these hyperparameters exceed their optimal values. This deterioration is related to the gradual loss of generalization for higher values of $m$ and $\alpha_{max}$. Deterioration for EnsOR is much slower, which means that this approach is more resistant to overtraining.

Table \ref{tab1} shows optimal hyperparameters, test and training errors and compares the sensitivities of ensembling variants. The lowest values are in bold. Table  \ref{tab1} clearly shows that EnsOR performs best. This method gave the lowest errors for each dataset and had the lowest sensitivity to both hyperparameters. By contrast, EnsR performed worst in terms of error and sensitivity. Note that the optimal hyperparameter values are smaller for EnsR and EnsCT than for EnsOR. This means that the base models in these boosting variants need to be less flexible (weaker) to prevent overfitting. Nevertheless, EnsR and EnsCT do not perform as well as EnsOR.

\begin{table}[htbp]

  \begin{center}
\setlength{\tabcolsep}{4pt}
  \caption{Optimal RandNN hyperparameters, errors and ensemble sensitivity.} \label{tab1}
    \begin{tabular}{llcccccc}
    \toprule
      Data & Ensemble & $m^*$ & $\alpha_{max}^*$ & $MAPE_{tst}$ & $MAPE_{trn}$ & $S_m$ & $S_{\alpha_{max}}$ \\
    \midrule
    \textbf{PL}
     & EnsR  & 20    & 40               & 1.18         & 0.76         & 0.641 & 0.436              \\
     & EnsCT & 20    & 60               & 1.15         & \textbf{0.72}         & 0.092 & 0.174              \\
     & EnsOR & 40    & 70               &\textbf{ 1.14}         &\textbf{ 0.72}         & \textbf{0.064} & \textbf{0.110}              \\
    \midrule
    \textbf{GB}
     & EnsR  & 20    & 10               & 2.55         & 1.55         & 0.509 & 1.531              \\
     & EnsCT & 10    & 50               & 2.52         & \textbf{1.51}         & \textbf{0.104} & 0.628              \\
     & EnsOR & 20    & 50               & \textbf{2.49}         & \textbf{1.51}         & \textbf{0.104} & \textbf{0.163}              \\
   \midrule
    \textbf{FR}
     & EnsR  & 10    & 10               & 1.65         & 1.27         & 0.466 & 0.853\\
     & EnsCT & 10    & 20               & 1.61         & 1.24         & 0.088 & 0.436\\
     & EnsOR & 20    & 40               & \textbf{1.57}         &\textbf{ 1.20}         & \textbf{0.058} & \textbf{0.069}\\
    \midrule
    \textbf{DE}
     & EnsR  & 10    & 40               & 1.29         & 1.37         & 0.774 & 0.572              \\
     & EnsCT & 10    & 70               & 1.21         & 1.26         & 0.138 & 0.141              \\
     & EnsOR & 40    & 80               & \textbf{1.17}         & \textbf{1.05}         & \textbf{0.036} & \textbf{0.063} \\
    \bottomrule
    \end{tabular}%
  \label{tab1}%
  \end{center}
\end{table}%

Fig. \ref{figk} demonstrates test MAPE in successive steps of ensembling. We can observe from this figure that for EnsOR the error converges faster with $k$ than for other ensemble variants. The convergence curve is also smoother for EnsOR than for its competitors. For them, in many cases, adding a new member to the ensemble
causes a temporary increase in error. Moreover, it was observed for EnsR that if the hyperparameters are too high (greater than optimal), which means a flexible base model, the error starts to successively increase with subsequent ensembling stages. This phenomenon is to a lesser degree observed for EnsCT, but not observed for EnsOR. Fig. \ref{figk2} demonstrates this problem for PL data and $m=100$, $\alpha_{max}=70$.

\begin{figure}[h]
	\centering
	\includegraphics[width=0.5\textwidth]{PLk2.eps}
    \caption{Ensemble error at successive stages for too flexible base models.}
    \label{figk2}
\end{figure}

To improve further ensemble learning based on opposed response, we
introduce weights for training patterns which express their similarity to the query pattern. The more similar training pattern $\mathbf{x}_i$ to query pattern $\mathbf{x}$, the higher the weight. The similarity measure is a scalar product, $\mathbf{x}^T\mathbf{x}_i$. The opposed response takes the form:

\begin{equation}
\hat{\mathbf{y}}'_{k,i}=\mathbf{y}_i+w_i\mathbf{r}_{k,i},   i = 1, ..., N
\label{v1}
\end{equation}
where $\mathbf{r}_{k,i}=\mathbf{y}_i-F_k(\mathbf{x}_i)$ is a residual vector for the $i$-th training pattern and $w_i \in [0, 1]$ is the weight of this pattern.

In the base variant of EnsOR, the weights for all patterns are equal to one. In typical ensemble learning, the weights are zero (the base models at each stage of ensembling learn on the original training set $\Phi$; such an approach we considered in \cite{Dud21}). By introducing weights, we try to balance these two approaches.

As a weighting function, we consider four variants. In the simplest one, EnsOR1, we assume that the weighting function $g$ is just the scalar product:

\begin{equation}
g(\mathbf{x},\mathbf{x}_i) =\mathbf{x}^T\mathbf{x}_i
\label{sp}
\end{equation}

To avoid negative values of \eqref{sp} we can also use $g(\mathbf{x},\mathbf{x}_i) =\frac{1}{2}(\mathbf{x}^T\mathbf{x}_i+1)$ or replace negative values with zeros.

In the second variant, EnsOR2, we sort the training patterns according to similarity to the query pattern, from the most to the least similar. Let $r=1, ..., N$ be the rank of the training patterns in the similarity ranking. The weighting function expresses the linear dependence of the weight on the rank:

\begin{equation}
g(\mathbf{x},\mathbf{x}_i) =1+\frac{1-r_i}{N}
\label{rr}
\end{equation}

In the third variant, EnsOR3, the weighting function expresses the non-linear dependence of the weight on the rank:

\begin{equation}
g(\mathbf{x},\mathbf{x}_i) =\left(1+\frac{1-r_i}{N}\right)^d
\label{rr}
\end{equation}
where $d>1$ (we assume $d=4$).

In the fourth variant, EnsOR4, the most similar training patterns to the query pattern have unity weights, while the others have zero weights:

\begin{equation}
g(\mathbf{x},\mathbf{x}_i) =
\begin{cases}
    1       & \quad \text{if } \mathbf{x}_i \in \Xi_\kappa(\mathbf{x})\\
    0  & \quad \text{if } \mathbf{x}_i \notin \Xi_\kappa(\mathbf{x})
  \end{cases}
\label{nn}
\end{equation}
where $\Xi_\kappa(\mathbf{x})$ denotes a set of $\kappa$ nearest neighbors of query pattern $\mathbf{x}$ in $\Phi$.

An example of weights assigned to the training patterns by the above weighting functions are shown in Fig. \ref{figw}.

\begin{figure}[h]
	\centering
	\includegraphics[width=0.5\textwidth]{Weights.eps}
    \caption{Examples of weights assigned to training pattern in different weighing variants.}
    \label{figw}
\end{figure}

\begin{figure}[h]
	\centering
	\includegraphics[width=0.24\textwidth]{PLnn.eps}
    \includegraphics[width=0.24\textwidth]{GBnn.eps}
    \includegraphics[width=0.24\textwidth]{FRnn.eps}
    \includegraphics[width=0.24\textwidth]{DEnn.eps}
    \caption{Ensemble error for EnsOR4 depending on $\kappa$.}
    \label{fignn}
\end{figure}

Fig. \ref{fignn} allows us to evaluate the impact of the number of nearest neighbours $\kappa$ in EnsOR4 on the test MAPE. The optimal value of $\kappa$ depends on the data set: for PL $\kappa=14$, for GB $\kappa=40$, for FR $\kappa=38$, and for DE $\kappa=8$.

\begin{table}[htbp]

  \begin{center}
\setlength{\tabcolsep}{4pt}
  \caption{Results for EnsOR with weighted patterns.} \label{tab1}
    \begin{tabular}{lcccccc}
    \toprule
      Data & EsnOR1 & EnsOR2 & EnsOR3 & EnsOR4 & RandNN \cite{Dud21} & Ens1 \cite{Dud21}  \\
    \midrule
    \textbf{PL}
     & 1.1419 & 1.1381 & 1.1329 & \textbf{1.1295} & 1.3206 & 1.1417  \\
    \textbf{GB}
     & 2.4935 & 2.4804 & 2.4822 & \textbf{2.4803} & 2.6126 & 2.5148  \\
    \textbf{FR}
     & 1.5668 & 1.5570 & 1.5524 & \textbf{1.5492} & 1.6711 & 1.5690  \\
    \textbf{DE}
     & 1.1743 & 1.1711 & 1.1683 &\textbf{1.1655} & 1.3809 & 1.1811  \\
    \bottomrule
    \end{tabular}%
  \label{tab2}%
  \end{center}
\end{table}%

Table \ref{tab2} compares the proposed methods of pattern weighting. In this table results for individual RandNN and Ens1 are also shown. Ens1 is a classical (not boosted) ensemble of RandNN. Ens1 constructs the final forecast as an average of the responses of RandNN, which learn simultaneously on the same training set $\Phi$ \cite{Dud21}. As can be seen from this table, EnsOR4 outperformed its competitors as well as classical ensembling Ens1. The much higher error for individual RandNN justifies fully ensembling. It is worth mentioning that comparing Ens1 with other forecasting models, including statistical models (ARIMA, exponential smoothing, Prophet) and machine learning models (MLP, SVM, ANFIS, LSTM, GRNN, nonparametric models), reported in \cite{Dud21} clearly shows that Ens1 outperforms all its competitors in terms of accuracy.

\section{Conclusion}

Forecasting complex time series with multiple seasonalities is a challenging problem, but one we solve using ensemble of randomized NNs. We propose three methods of boosting the RandNN ensemble. We showed that in ensemble learning based on residuals, the base models have different tasks to solve at successive stages of ensembling. Thus, the base model which is optimal at a given stage may not be optimal at other stages. To avoid having to select an optimal model at each stage of ensembling, which is unreasonable and too time-consuming, we propose to unify the tasks solved at all stages. Doing so allows us to use the same base model (the same architecture and hyperparameters), RandNN in our case, which is optimal for all stages. To unify the tasks, we propose ensemble learning based on corrected targets and ensemble learning based on opposed response. The latter proved to be more resistant to task degradation at subsequent stages.

The experimental studies performed on four forecasting problems expressing triple seasonalities confirmed that the opposed response-based approach outperforms its competitors in terms of forecasting accuracy as well as sensitivity to both the base model hyperparameters and ensemble size. Further improvement of the winning solution was achieved by weighting the training patterns according to their similarity to the query pattern.

In further research, we plan to develop the opposed response-based approach for other types of learners, e.g. decision trees, and other types of problems (regression, classification).

\section{Introduction}

The power system load is a nonlinear and nonstationary process that can change rapidly due to many factors such as macroeconomic variations, weather, electricity prices, consumer types and habits, etc. Therefore, electricity demand forecasting, which is essential for the power system operation and planning, is a big challenge. In this study we consider mid-term electrical load forecasting (MTLF) focusing on monthly electricity demand forecasting over 12 months horizon.

MTLF methods can be roughly classified into statistical/econometrics methods or ML methods \cite{Sug11}. The former include ARIMA, ETS and linear regression (LR). ARIMA and ETS can deal with seasonal TS but LR requires additional operations such as decomposition or extension of the model with periodic components \cite{Bar01}. Limited adaptability of the statistical MTLF models and problems with nonlinear relationship modeling have increased researchers' interest in ML and AI tools \cite{Gon08}. Of these, neural networks (NNs) are the most popular because of their attractive features including learning capabilities, universal approximation property, nonlinear modeling and massive parallelism. Some examples of using NNs for MLTF are: \cite{Chen17} where NN uses historical loads and weather variables to predict monthly demand and is trained by heuristic algorithms to improve performance, \cite{Gav01} where Kohonen NN is used, \cite{Dov99} where NNs are supported by fuzzy logic, \cite{Pel19} where generalized regression NN is used, \cite{Pei11} where weighted evolving fuzzy NN is used, and \cite{Ahm19} where NNs, LR and AdaBoost are combined.

Recent trends in ML such as deep recurrent NNs (RNNs), are very attractive for TS forecasting \cite{Hew19}. RNNs are able to exhibit temporal dynamic behavior using their internal state to process sequences of inputs. Recent works have reported that RNNs, such as the LSTM, provide high accuracy in forecasting and outperform most of the traditional statistical and ML methods \cite{Yan18}. Some application examples of LSTMs to load forecasting can be found in \cite{Bed18,Zhe17,Nar17}.

In \cite{Dud20} we proposed a hybrid residual dilated LSTM and ETS model (ETS+RD-LSTM) for MTLF. This model was based on the winning submission to the M4 forecasting competition 2018 \cite{Mar18a}, developed by Slawek Smyl \cite{Smy20}. A simulation study confirmed the high performance of the model and its competitiveness with classical models such as ARIMA and ETS as well as state-of-the-art ML models. In this work we extend ETS+RD-LSTM by introducing initial TS normalization, i.e. detrending and unifying the variance. This method of TS preprocessing we used in our previous works achieving very good results \cite{Dud20a}. Recently we used it for LSTM model obtaining a 15\% reduction in error \cite{Pel20}. We expect that the TS initial normalization, which simplifies the relationship between input and output data, allows ETS+RD-LSTM to improve its performance.

\section{Forecasting Model}

The proposed model is a modified version of ETS+RD-LSTM which we described in \cite{Dud20}. We extend ETS+RD-LSTM by introducing initial TS normalization. A normalization procedure removes a trend and unifies variance of the TS. The normalized TS exhibit yearly patterns which are further removed using deseasonalization as an integral part of ETS+RD-LSTM. The normalized and deseasonalized TS are forecasted using RD-LSTM. Then the forecasts are reseasonalized using seasonal components extracted by ETS. To reduce the model variance, we use ensembling at three levels which aggregates individual forecasts. The resulting aggregated forecasts are finally denormalized. For denormalization, the forecasts of the mean yearly demand and yearly dispersion are needed. They are produced by additional two ETS modules.

\subsection{Architecture and Features}

An architecture of the proposed forecasting system is shown in Fig. \ref{figA}. The system components are as follows (symbols in italics denote the sets of TS or forecasts):
\begin{itemize}
	\item
	Normalization -- each original monthly electricity demand TS is normalized. This procedure removes a trend from the TS and unifies its variance. Normalization module loads a set of TS ($Z$), calculates the series of yearly mean demands ($\overline Z$) and yearly dispersions ($\Sigma$) for each TS, and determines normalized series ($Y$).

	\item
	ETS -- exponential smoothing modules for forecasting the yearly mean demands and their dispersions. These values are necessary for denormalization.
	\item
	Deseasonalization -- each normalized TS is deseasonalized. This procedure extracts the seasonal components, $S$, individually for each series using ETS (ETSd module), and determines deseasonalized TS, $X$.

	\item
	RD-LSTM -- residual dilated LSTM for forecasting the normalized and deseasonalized TS, $X$.

	\item
	Reseasonalization -- each TS forecast produced by RD-LSTM is reseasonalized using inverse operations to deseasonalization.

	\item
	Ensembling -- the reseasonalized forecasts produced by RD-LSTM are averaged. The ensembling module receives the sets of individual forecasts, $\hat{Y}_k^r$, and returns an aggregated forecast for each TS, $\hat{Y}_{\text{avg}}$.

	\item
	Denormalization -- the averaged forecasts $\hat Y_{\text{avg}}$ are denormalized using forecasted values of the yearly means, $\hat{\overline{Z}}$, and dispersions, $\hat{\Sigma}$.

\end{itemize}

\begin{figure
	\centering
	\includegraphics[width=0.9\textwidth]{schemat2.png}
	\caption{The proposed forecasting system architecture}
	\label{figA}
\end{figure}

The proposed system has a hybrid and hierarchical structure. It combines statistical modeling (ETS), advanced ML (RD-LSTM), and ensembling. ETS is used as a forecasting model for yearly means, $\overline Z$, and dispersions, $\Sigma$, as well as for extraction of seasonal components (ETSd). The preprocessed TS, without trend and seasonal variations, are forecasted using RD-LSTM. Details of data preprocessing and flow are described in subsection 2.2.

The TS are exploited in a hierarchical manner, meaning that both local and global components are utilized in order to extract and combine information at either a series or a dataset level, thus enhancing the forecasting accuracy. The global features are learned by RD-LSTM across many TS (cross-learning). The specific features of each individual TS, such as trend, variance, and seasonality, are extracted by normalization and ETSd modules. Thus, each series has a partially unique and partially shared model.

The strength of RD-LSTM, which revealed in M4 competition, is cross-learning, i.e., using many series to train a single model. This is unlike standard statistical TS algorithms, where a separate model is developed for each series. Another important ingredient in the success of the proposed method precursor in the M4 competition was the on-the-fly preprocessing that was an inherent part of the training process. Crucially, the parameters of this preprocessing (in the proposed model these are twelve initial seasonal components and smoothing coefficient $\beta$, see subsection 2.2) were being updated by the same overall optimization procedure (SGD) as weights of RD-LSTM, with the overarching goal of minimizing forecasting errors. This enables the model to simultaneous optimization of data representation, i.e. searching for the most suitable representations of input and output data for RD-LSTM, and forecasting performance.

ETSd is used as the preprocessing tool. It extracts a seasonal component which is used for deseasonalization of the normalized TS. ETSd  was inspired by the Holt-Winters multiplicative seasonal model. However, it has been simplified by removing trend and level components (see subsection 2.2). This is because the input TS are normalized, i.e. they have no trend and their level is one. ETSd is optimized simultaneously with RD-LSTM using pinball loss function \cite{Smy20}:

\begin{equation}
L_t =
\begin{cases}
(x_t-\hat{x}_t)\tau       & \text{if } x_t \geq \hat{x}_t\\
(\hat{x}_t-x_t)(1-\tau)  &\text{if } \hat{x}_t > x_t
\end{cases}
\label{eqLt}
\end{equation}
where $x_t$ and $\hat{x}_t$ are the actual and forecasted values, respectively, and $\tau \in (0, 1)$ is a parameter controlling the loss function asymmetry.

When $\tau=0.5$ the loss function is symmetrical and penalizes positive and negative deviations equally. When the model tends to have a positive or negative bias, we can reduce the bias by introducing $\tau$ smaller or larger than $0.5$, respectively. Thus, the asymmetric pinball loss function, penalizing positive and negative deviations differently, allows the method to deal with bias.

ETS for forecasting the yearly mean demands and their dispersions are defined as innovations state space models \cite{Hyn08}. They combine the seasonal, trend and error components in different ways (additively or multiplicatively). For each TS the optimal ETS model is selected using Akaike information criterion (AIC).

Ensembling is used for reduction the model variance related to the stochastic nature of SGD, and also related to data and parameter uncertainty. Ensembling is seen as a much more powerful regularization technique than more popular alternatives, e.g. dropout or L2-norm penalty \cite{Ore20}. In our case, ensembling combines individual forecasts at three levels: stage of training level, data subset level and model level. At the stage of training level, the forecasts produced by $L$ most recent training epochs are averaged. This can reduce the effect of stochastic searching, i.e. calming down the noisy SGD optimization process. At the data subset level, we use $K$ models which learn on the subsets of the training set, $\Psi_1$, $\Psi_2$, ..., $\Psi_K$. Each $k$-th model produces forecasts for TS included in its own training subset $\Psi_k$. Then the forecasts  produced by the pool of $K$ models are averaged individually for each TS. The third level of ensembling simply averages the forecasts for each TS generated in $R$ independent runs of a pool of $K$ models. In each run, the training subsets $\Psi_k$ are created anew (see \cite{Dud20} for details).

\subsection{Time Series Processing}

A monthly electricity demand TS exhibits a trend, yearly seasonality and random component (see Fig. \ref{figP}(a)). To simplify the forecasting problem, the TS is preprocessed as follows. Let $\{z_t\}_{t=1}^N$ be a monthly electricity demand TS starting from January and ending in December. This TS is divided into yearly subsequences $\{z_t^i\}_{t=12(i-1)+1}^{12(i-1)+12}, i=1, ..., N/12$. Each $i$-th subsequence is expressed by a vector $\mathbf{z}_i = [z_{i,1} z_{i,2} … z_{i,12}]^T$. The normalized version of $\mathbf{z}_i$, $\mathbf{y}_i = [y_{i,1} y_{i,2} … y_{i,12}]^T$, is determined as follows:

\begin{equation}\label{eqy}
y_{i,j} = \frac{z_{i,j}-\overline{z}_i}{\sigma_i}+1
\end{equation}
where $j = 1, ..., 12$, $\overline{z}_i$ is a mean of subsequence $\{z_t^i\}$, and $\sigma_i = \sqrt{\sum_{j=1}^{12} (z_{i,j}-\overline{z}_i)^2}$ is a measure of its dispersion.

\begin{figure
	\centering
	\includegraphics[width=0.32\textwidth]{z.eps}
	\includegraphics[width=0.32\textwidth]{Sr.eps}
	\includegraphics[width=0.32\textwidth]{sig.eps}
		\includegraphics[width=0.32\textwidth]{y.eps}
	\includegraphics[width=0.32\textwidth]{s.eps}
	\includegraphics[width=0.32\textwidth]{x.eps}
	\caption{TS preprocessing: (a) original TS $\{z_t\}$, (b) yearly mean demand TS $\{\overline{z}_t\}$, (c) yearly dispersion TS $\{\sigma_t\}$, (d) normalized TS $\{y_t\}$, (e) seasonal component TS $\{s_t\}$, and (f) normalized and deseasonalized TS $\{x_t\}$}
	\label{figP}
\end{figure}

Note that yearly subsequences $\{z_t^i\}$ have different means and dispersions (see Fig. \ref{figP}(a)). After normalization they are unified, i.e. all yearly subsequences have an average of one, the same variance and also a unity of length. They carry information about the shapes of the yearly sequences. Now we create a new TS composed of normalized subsequences representing successive yearly periods: $\{y_t\} = \{\mathbf{y}_i^T\}_{i=1}^{N/12} = \{y_{1,1},y_{1,2}, ..., y_{N/12,12}\}$. This TS is shown in Fig. \ref{figP}(d). Note its regular character and stationarity.

The TS of the mean yearly demand, $\{\overline{z}_i\}_{i=1}^{N/12}$, and yearly dispersion, $\{\sigma_i\}_{i=1}^{N/12}$, are shown in Fig. \ref{figP}(b) and (c), respectively. They are forecasted by ETS one step ahead (for the next year) and used for denormalization.

The normalized TS, $\{y_t\}$, is further deseasonalized. To do so, we use a simplified Holt-Winters multiplicative seasonal model with only one component:

\begin{equation}
\begin{aligned}
s_{t+12}=\beta y_t + (1-\beta)s_t
\label{eqls}
\end{aligned}
\end{equation}
where $s_t$ is the seasonal component at timepoint $t$ and $\beta \in [0, 1]$ is a smoothing coefficient.

The seasonal component is shown in Fig. \ref{figP}(e). It is used for deseasonalization during the on-the-fly preprocessing. The TS $\{y_t\}$ is deseasonalized in each training epoch using the updated values of seasonal components. These updated values are calculated from \eqref{eqls}, where  parameters, 12 initial seasonal components and $\beta$, are increasingly fine tuned for each TS in each epoch by SGD.

The TS is deseasonalized using rolling windows: input and output ones. The input window contains twelve consecutive elements of the TS which after deseasonalization will be the RD-LSTM inputs. The corresponding output window contains the next twelve consecutive elements, which after deseasonalization will be the RD-LSTM outputs. The TS fragments inside both windows are deseasonalized by dividing them by the relevant seasonal component. Then, to limit the destructive impact of outliers on the forecasts, a squashing function, log(.), is applied. The resulting deseasonalization can be expressed as follows:

\begin{equation}
x_t=\log \left(\frac{y_t}{s_t} \right)
\label{eqxt}
\end{equation}
where $x_t$ is the deseasonalized $t$-th element of the normalized TS, and $s_t$ is the $t$-th seasonal component.

The preprocessed TS sequences contained in the successive input and output windows are represented by vectors: $\textbf{x}_t^{in} = [x_t \dots x_{t+12}]$, $\textbf{x}_t^{out} = [x_{t+13} \dots x_{t+24}]$, $t = 1, ..., N-24$. These vectors are included in the training subset for the $i$-th TS: $\Phi_i=\{(\textbf{x}_t^{in}, \textbf{x}_t^{out})\}_{t = 1}^{N-24}$. The training subsets for all $M$ TS are combined and form the training set $\Psi=\{\Phi_1, ..., \Phi_M\}$ which is used for RD-LSTM cross-learning. Note the dynamic character of the training set. It is updated in each epoch because the seasonal components in \eqref{eqxt} are updated by SGD.

The forecasts produced by RD-LSTM, $\hat{x}_t$, are reseasonalized as follows:

\begin{equation}
\hat{y}_t=s_t \exp (\hat{x}_t)
\label{eqyt}
\end{equation}
where $s_t$ is determined from \eqref{eqls} on the basis of the TS history.

Finally, the TS is denormalized using transformed equation \eqref{eqy}:

\begin{equation}\label{eqzt}
\hat{z}_{i,j} = (\hat{y}_{i,j}-1)\hat{\sigma}_i + \hat{\overline{z}}_i
\end{equation}
where $i$ refers to the forecasted yearly period, $j = 1, ..., 12$, $\hat{\overline{z}}_i$ and $\hat{\sigma}_i$ are the forecasted yearly mean and dispersion for period $i$.

\subsection{Residual Delated LSTM}

The RD-LSTM architecture used in this study is shown in Fig. \ref{figA}(a) \cite{Dud20}. It is composed of four recurrent layers and a linear unit LU. The first layer consists of the standard LSTM block shown in Fig. \ref{figA}(b). The subsequent three layers consist of RD-LSTM blocks, i.e. blocks equipped with dilated recurrent skip connections and a spatial shortcut path from lower layers (Fig. \ref{figA}(c)).

A standard LSTM block consists of hidden state $\textbf{h}_t$ and cell state $\textbf{c}_t$. The cell state contains information learned from the previous time steps which can be added to or removed from the cell state using the gates: input gate ($i$), forget gate ($f$) and output gate ($o$). At each time step $t$, the block uses the past state, $\textbf{c}_{t-1}$ and $\textbf{h}_{t-1}$, and input $\textbf{x}_t$ to compute output $\textbf{h}_{t}$ and updated cell state $\textbf{c}_{t}$. The hidden and cell states are recurrently connected back to the block input. All of the gates are controlled by the hidden state of the past cycle and input $\textbf{x}_t$. The equations for a standard LSTM block are shown in Tab. \ref{tab1}.

\begin{figure}[t]
	\centering
	\includegraphics[width=0.99\textwidth]{RD-LSTM.png}
	\caption{RD-LSTM architecture (a), LSTM block (b), and RD-LSTM block (c)}
	\label{figA}
\end{figure}

\begin{table}[t]
	\caption{Equations for the forward pass of LSTM blocks}
	\label{tab1}
	\setlength{\tabcolsep}{20pt}
	\centering
	\begin{tabular}{|l|l|}
		\hline
		Standard LSTM block & RD-LSTM block\\
		\hline
		$\textbf{f}_t^1= \sigma_g(\textbf{W}_f^1 \textbf{x}_t+\textbf{V}_f^1\textbf{h}_{t-1}^1+ \textbf{b}_f^1)$ & $\textbf{f}_t^l= \sigma_g(\textbf{W}_f^l \textbf{h}_t^{l-1}+\textbf{V}_f^l\textbf{h}_{t-d}^l+ \textbf{b}_f^l)$\\
		$\textbf{i}_t^1=\sigma_g(\textbf{W}_i^1\textbf{x}_t+\textbf{V}_i^1\textbf{h}_{t-1}^1+ \textbf{b}_i^1)$ & $\textbf{i}_t^l= \sigma_g(\textbf{W}_i^l\textbf{h}_t^{l-1}+\textbf{V}_i^l\textbf{h}_{t-d}^l+ \textbf{b}_i^l)$\\
		$\textbf{g}_t^1=\sigma_c(\textbf{W}_g^1\textbf{x}_t+\textbf{V}_g^1\textbf{h}_{t-1}^1+\textbf{b}_g^1)$ & $\textbf{g}_t^l=\sigma_c(\textbf{W}_g^l\textbf{h}_t^{l-1}+\textbf{V}_g^l\textbf{h}_{t-d}^l+\textbf{b}_g^l)$\\
		$\textbf{o}_t^1=\sigma_g(\textbf{W}_o^1\textbf{x}_t+\textbf{V}_o^1\textbf{h}_{t-1}^1+\textbf{b}_o^1)$ & $\textbf{o}_t^l=\sigma_g(\textbf{W}_o^l\textbf{h}_t^{l-1}+\textbf{V}_o^l\textbf{h}_{t-d}^l+\textbf{b}_o^l)$\\
		$\textbf{c}_t^1 = \textbf{f}_t^1  \otimes  \textbf{c}_{t-1}^1  + \textbf{i}_t^1 \otimes \textbf{g}_t^1$ & $\textbf{c}_t^l = \textbf{f}_t^l  \otimes  \textbf{c}_{t-d}^l  + \textbf{i}_t^l \otimes \textbf{g}_t^l$\\
		$\textbf{h}_t^1 = \textbf{o}_t^1   \otimes  \sigma_c(\textbf{c}_t^1) $ & $\textbf{h}_t^l = \textbf{o}_t^l   \otimes  (\sigma_c(\textbf{c}_t^l) + \textbf{h}_t^{l-1})$\\
		\hline
	\end{tabular}
	\vspace{1ex}

	{\raggedright
		\tiny
	where $\textbf{W}$, $\textbf{V}$ and $\textbf{b}$ are input weights, recurrent weights and biases, respectively, $\sigma_c$ is a hyperbolic tangent function, $\sigma_g$ is a sigmoid activation function $(1+e^{-x})^{-1}$, $\otimes$ denotes the Hadamard product, superscript 1 refers to the first layer of RD-LSTM network, where we use the standard LSTM block, superscript $l$ indicates the layer number for RD-LSTM blocks (from 2 to 4 in our case), and $d$ is a dilation (3, 6 or 12 in our case). \par}
\end{table}

The RD-LSTM blocks employ dilation mechanism proposed in \cite{Cha17}. It is to solve three main problems related to RNN learning on long sequences: complex dependencies, vanishing and exploding gradients, and efficient parallelization. It is characterized by multi-resolution dilated recurrent skip connections. To compute the current states of the LSTM block, the last $d-1$ states are skipped, i.e. a dilated LSTM block receives as input states $\textbf{c}_{t-d}$ and $\textbf{h}_{t-d}$, where $d>1$ is a dilation. Usually multiple dilated recurrent layers are stacked with hierarchical dilations to construct a system, which learns the temporal dependencies of different scales at different layers. In \cite{Cha17}, it was shown that this solution can reliably improve the ability of recurrent models to learn long-term dependency. Dilated RNN can be particularly useful for  seasonal TS. In this case  dilations can be related to seasonality. In our case we use $d=$ 3, 6 and 12.

A residual LSTM was proposed in \cite{Kim17} to enable effective training of deep networks with multiple LSTM layers by avoiding vanishing or exploding gradients in the temporal domain. Residual LSTM provides a shortcut path between adjacent layer outputs. The shortcut paths are used to allow gradients to flow through a network directly, without passing through non-linear activation functions. In our implementation, we introduced shortcut paths extending equation for the hidden state (note additional component, $\textbf{h}_t^{l-1}$, for a hidden state in the right column of Table \ref{tab1}, where the RD-LSTM computation process is shown).

A linear unit, LU, transforms the output of the last layer, $\textbf{h}_t^4$, into the forecast of the output x-vector:

\begin{equation}
\hat{\textbf{x}}_t^{out} = \textbf{W}_x\textbf{h}_{t}^4+\textbf{b}_x
\label{eqL7}
\end{equation}

Note that RD-LSTM works on 12-component x-vectors. It produces the forecasts for the whole yearly period receiving the previous yearly period as input. The parameters of RD-LSTM, i.e. input weights $\textbf{W}$, recurrent weights $\textbf{V}$, and biases $\textbf{b}$, are learned using SGD in the cross-learning mode simultaneously with the ETSd parameters. The length of the cell and hidden states, $m$, the same for all layers, was selected on the training set to ensure the highest performance.

\section{Results}

The proposed forecasting model is applied for monthly electricity demand forecasting for 35 European countries. The real-world data are taken from the ENTSO-E repository (www.entsoe.eu). The TS lengths vary from 5 to 24 years. The forecasting problem is to produce the forecasts for the twelve months of 2014 (last year of data) using data from the previous period for training. For hyperparameter selection the model learned on the TS fragments up to 2012, and then it was validated on 2013. The selected hyperparameters were used to build the model for 2014: number of epochs $10$, learning rate $10^{-3}$, length of the cell and hidden states $m=40$, asymmetry parameter in pinball loss $\tau=0.4$, ensembling parameters: $L=5$, $K=4$, $R=3$.
RD-LSTM was implemented in C++ relying on the DyNet library and run in parallel on an 8-core CPU. We employ R implementation of ETS (function \texttt{ets} from package \texttt{forecast}).

The proposed model was compared with its predecessor, ETS+RD-LSTM \cite{Dud20}, and other state-of-the-art models based on ML as well as classical statistical models. They include: $k$-nearest neighbor weighted regression model, $k$-NNw, fuzzy neighborhood model, FNM, general regression NN model, GRNN, multilayer perceptron, MLP, adaptive neuro-fuzzy inference system, ANFIS, LSTM model, ARIMA model, and ETS model. All ML models were used also in +ETS versions, where the TS were initially normalized and the yearly mean and dispersion were forecasted using ETS (just like in this study). Details of the comparative models can be found in \cite{Dud20a,Pel19b,Pel18,Pel20}.

Table \ref{tab2} shows the forecast results averaged over 35 countries, i.e. median of absolute percentage error (APE), mean APE (MAPE), interquartile range of APE as a measure of the forecast dispersion, root mean square error (RMSE), and mean PE (MPE). The proposed model is denoted by 3ETS+RD-LSTM. As can be seen from this table, all error measures indicate that 3ETS+RD-LSTM is the most accurate model comparing with its competitors. It outperforms its predecessor, ETS+RD-LSTM, by 8.7\% in MAPE and 9.5\% in RMSE.

\begin{table}[]

	\caption{Results comparison among proposed and comparative models}
	\label{tab2}
	\setlength{\tabcolsep}{8pt}
	\centering
	\begin{tabular}{|l|c|c|c|c|c|}
		\hline

		Model       & Median \textit{APE} & \textit{MAPE} & \textit{IQR} & \textit{RMSE} & \textit{MPE} \\ \hline

		k-NNw        & 2.89 & 4.99 & 3.85 & 368.79 & -1.87 \\
		FNM          & 2.88 & 4.88 & 4.26 & 354.33 & -2.03 \\
		N-WE         & 2.84 & 5.00 & 3.97 & 352.01 & -1.91 \\
		GRNN         & 2.87 & 5.01 & 4.02 & 350.61 & -1.87 \\
		k-NNw+ETS    & 2.71 & 4.47 & 3.52 & 327.94 & -1.25 \\
		FNM+ETS      & 2.64 & 4.40 & 3.46 & 321.98 & -1.26 \\
		N-WE+ETS     & 2.68 & 4.37 & 3.36 & 320.51 & -1.26 \\
		GRNN+ETS     & 2.64 & 4.38 & 3.51 & 324.91 & -1.26 \\
		MLP          & 2.97 & 5.27 & 3.84 & 378.81 & -1.37 \\
		MLP+ETS      & 3.11 & 4.80 & 4.12 & 358.07 & -1.71 \\
		ANFIS        & 3.56 & 6.18 & 4.87 & 488.75 & -2.51 \\
		ANFIS+ETS    & 3.54 & 6.32 & 4.26 & 464.29 & -1.30 \\
		LSTM         & 3.73 & 6.11 & 4.50 & 431.83 & -3.12 \\
		LSTM+ETS     & 3.08 & 5.19 & 4.54 & 366.45 & -1.41 \\
		ARIMA        & 3.32 & 5.65 & 5.24 & 463.07 & -2.35 \\
		ETS          & 3.50 & 5.05 & 4.80 & 374.52 & -1.04 \\
		ETS+RD-LSTM  & 2.74 & 4.48 & 3.55 & 347.24 & -1.11 \\
		3ETS+RD-LSTM & 2.64 & 4.09 & 3.13 & 314.01 & -0.32	\\ 	 \hline
	\end{tabular}
\end{table}

MPE provides information on potential forecast bias. All the models produced negatively biased forecasts, i.e. overpredicted.
But for 3ETS+RD-LSTM, the $t$-test did not reject the null hypothesis that PE comes from a normal distribution with mean equal to zero ($p$-value = 0.44). All other models did not pass this test. So it can be concluded that 3ETS+RD-LSTM, as the only model, produced unbiased forecasts. Note that 3ETS+RD-LSTM has the mechanism to deal with bias. The loss function \eqref{eqLt} asymmetry is controlled by parameter $\tau$. It was selected as 0.4, which allowed the model to reduce the negative bias.

Fig. \ref{figC} depicts more detailed results, MAPE for each country. As can be seen, in most cases 3ETS+RD-LSTM is one of the most accurate models. Fig. \ref{figR} depicts the model rankings based on MAPE and RMSE. They show average ranks of the models in the rankings for individual countries. Note the first position of 3ETS+RD-LSTM in both rankings.

\begin{figure}[t
	\centering
	\includegraphics[width=1\textwidth]{MAPE35.eps}
	\caption{MAPE for each country}
	\label{figC}
\end{figure}

\begin{figure}[t
	\centering
	\includegraphics[width=0.40\textwidth]{MAPErank.eps}
	\includegraphics[width=0.409\textwidth]{RMSErank.eps}
	\caption{Rankings of the models}
	\label{figR}
\end{figure}

Examples of forecasts produced by the selected models are shown in Fig. \ref{figPL}. For PL and DE data, MAPE is on the low level around 2\% while for GB data the forecasts are strongly underestimated, over 5\%. This is because the demand for GB went up unexpectedly in 2014 despite the downward trend observed in the previous period.

\begin{figure}[t
	\centering
	\includegraphics[width=0.37\textwidth]{PL.eps}
	\includegraphics[width=0.305\textwidth]{DE.eps}
	\includegraphics[width=0.305\textwidth]{GB.eps}
	\caption{Examples of forecasts produced by selected models}
	\label{figPL}
\end{figure}

Summarizing experimental research, it should be noted that the forecasting model performance depends significantly on the appropriate TS preprocessing. Although LSTM deals with raw data, without preprocessing \cite{Pel20}, introducing initial normalization and dynamic deseasonalization in 3ETS+RD-LSTM improved significantly LSTM performance.

It should be noted that LSTM based models are more complex than other comparative models. Due to the huge number of parameters and complicated learning procedure using backpropagation through time, the learning time of LSTM is much longer than for other comparative models.

\section{Conclusion}

In this work, we proposed an extended hybrid RD-LSTM and ETS model for MTLF. It combines initial TS decomposition into three components (normalized TS, trend, and dispersion), ETS modules for trend and dispersion forecasting, ETS for deseasonalization, advanced LSTM, and ensembling. The model has a hierarchical structure composed of a global part learned across many TS (LSTM) and a TS specific part (normalization and deseasonalization). Deseasonalization and LSTM are combined in a simultaneous learning process using SGD which leads to learning TS representations and mapping at the same time.

We used residual dilated LSTM, which can capture better long-term seasonal relationships and ensure more efficient training. This is because of dilated recurrent skip connections and a spatial shortcut path from lower layers. To deal with a forecast bias, an asymmetric pinball loss function was applied. Three-level ensembling provides regularization reducing the model variance, which has sources in the stochastic nature of SGD, and also in data and parameter uncertainty.

An experimental study, monthly electricity demand forecasting for 35 European countries, demonstrated the state-of-the-art performance of the proposed model. It generated more accurate forecasts than its predecessor (ETS+RD-LSTM), classical models such as ARIMA and ETS as well as state-of-the-art models based on ML.

\section{Introduction}

Ensemble methods are considered as a cornerstone of modern machine learning \cite{Ree18}. They are commonly used for regression and classification problems.
Ensembling is also very effective approach to increase the predictive power of forecasting models. Due to combining many base models, it improves the final forecasting accuracy as well as stability of the response when compared to the single model.
A success in ensemble learning depends on the proper flexibility of the ensemble members and the trade-off between their performance and diversity \cite{Bro05}. It is also determined by the way of generating learners at the successive stages of ensembling and the method of combining them.

The effectiveness of ensembling in forecasting is evidenced by the fact that in the most renowned forecasting competition, M4 competition \cite{Mar18}, of the 17 most accurate models, 12 used ensembling in some form \cite{Ati20}. The winning submission, which is a hybrid model combining exponential smoothing and long short-term memory, used three types of ensembling in the same time \cite{Smyl20}: combining results of the stochastic training process, bagging, and combining multiple runs.

To improve performance of the ensemble learning many approaches was proposed such as stacking \cite{Wol92}, bagging \cite{Bre96}, boosting \cite{Dru99}, negative correlation learning \cite{Chen09}, snapshot ensembles \cite{Hua17},
and horizontal and vertical ensembles developed for deep learning \cite{Xie13}. Boosting, that this work focuses on, is a general ensemble technique that involves sequentially adding base models to the ensemble where subsequent models correct the performance of prior models. This approach is very effective
as evidenced by the leading positions of the boosted models such as XGBoost \cite{Chen16}, i.e. ensemble of decision trees with regularized gradient boosting, in rankings of the winning models in competitions such as those organized by Kaggle. Boosting also has many applications in the forecasting field. Some examples are: \cite{Zie16}, where ensemble of boosted trees is used for bankruptcy prediction;
\cite{Ni20}, where XGBoost is combined with a Gaussian mixture model for monthly streamflow forecasting;
\cite{Li19}, where AdaBoost is applied as a component of the hybrid model for multi-step wind speed forecasting;
and \cite{Mit22}, where a natural gradient boosting algorithm is applied for solar power probabilistic forecasting.
This last work shows a valuable advantage of ensembling. It can produce probabilistic forecasts, i.e. the distribution of the forecasted variable in the future.

In this study, we propose a boosted version of our ensemble of randomized neural networks (RandNNs) for complex time series forecasting \cite{Dud21}. In \cite{Dud21}, we focused on the strategies for controlling diversity of ensemble members. The members were trained independently. Here, we construct an ensemble sequentially. When new member is added to the ensemble, it learns taking into account the results of the ensemble members so far. We consider three methods of boosting. The contribution of this study is threefold:

\begin{enumerate}
    \item
    We propose new methods of ensemble boosting: ensemble learning based on corrected targets and ensemble learning based on opposed response. They both are to ensure similar tasks for all ensemble members. This unification of the tasks justifies the use of identical base models in terms of architecture and hyperparameters at all stages of ensembling.

    \item
    We develop three ensemble learning approaches for forecasting complex time series with multiple seasonality. They are based on RandNNs, pattern-based time series representation and three methods of boosting: based on residuals, corrected targets and opposed response.

    \item
    We empirically compare the proposed boosted ensemble methods on the challenging short-term load forecasting problems with triple seasonality and conclude that the method based on opposed response outperforms its competitors in terms of forecasting accuracy and sensitivity to hyperparameters.

\end{enumerate}

The rest of the work is organized as follows. In Section II, we present a base model, RandNN. Details of the proposed three methods of boosted ensemble learning are described in Section III. An experimental framework used to evaluate and compare the proposed ensemble methods is described in Section IV. Finally, Section V concludes the work.

\section{Base Forecasting Model - RandNN}

As a base model we use a single hidden layer feedforward NN with $m$ logistic sigmoid hidden nodes \cite{Dud21a}. In randomized learning, the weights of hidden nodes are selected randomly from a uniform distribution and symmetrical interval $U=[-u,u]$. The biases of these nodes are calculated, according to recent research \cite{Dud19}, based on the weights as follows: $b_j = -\mathbf{a}_j^T\mathbf{x}^*_j$, where $\mathbf{a}_j$ is the vector of weights for the $j$-th hidden node, and $\mathbf{x}^*_j$ is one of the training patterns selected at random (see \cite{Dud19} and \cite{Dud20} for details, justification and other variants). This way o generating hidden nodes places the sigmoids into the input feature space limited to some hypercube, avoiding their saturated parts. Moreover, the sigmoids are distributed according to the data distribution. These improve aproximation and generalization abilities of the model \cite{Dud19}, \cite{Dud20}.

The hidden node sigmoids are combined linearly by the output nodes: $\varphi_k(\mathbf{x})=\sum_{j=1}^{m}\beta_{j,k}h_j(\mathbf{x})$, where $h_j(\mathbf{x})$ is the output of the $j$-th hidden node, and $\beta_{j,k}$ is the weight between $j$-th hidden and $k$-th output nodes. The only learnable parameters are the output weights. They are calculated from $\boldsymbol{\beta} = \mathbf{H}^+\mathbf{Y}$, where $\mathbf{H}$ is a matrix of the hidden layer outputs, $\mathbf{H}^+ $ denotes its Moore–Penrose generalized inversion, and $\mathbf{Y}$ is a~matrix of target output patterns. $\mathbf{H}$ is a nonlinear feature mapping from $n$-dimensional input space to $m$-dimensional projection space (usually $m \gg n$). Note that this projection is random. Due to the fixed parameters of hidden nodes, the optimization problem in randomized learning (selection of weights \boldsymbol{\beta}) becomes convex and can be easily solved by the standard least-squares method.

The forecasting model based on RandNN, which we adapt from \cite{Dud21a}, has two more components: encoder and decoder.
Let us consider time series $\{E_k\}_{k=1}^K$ with multiple seasonality. The encoder transforms this series into input and output patterns expressing seasonal sequences of the shortest length. The input patterns, $\mathbf{x}_i = [x_{i,1}, …, x_{i,n}]^T$, represent sequences $ \mathbf{e}_i = [E_{i,1},…, E_{i,n}]^T $, while the output patterns, $\mathbf{y}_i = [y_{i,1},…, y_{i,n}]^T$, represent forecasted sequences $ \mathbf{e}_{i+\tau} = [E_{i+\tau,1},…, E_{i+\tau,n}]^T $, where $n$ is a period of the seasonal cycle (e.g. 24 hours for daily seasonality), $i=1, ..., K/n$ is the sequence number, and $\tau \geq 1$ is a forecast horizon. The patterns are defined as follows:

\begin{equation}\label{eq1}
\mathbf{x}_i = \frac{\mathbf{e}_i-\overline{e}_i}{\widetilde{e}_i},
\qquad
\mathbf{y}_i = \frac{\mathbf{e}_{i+\tau}-\overline{{e}}_i}{\widetilde{{e}}_i}
\end{equation}
where $\overline{{e}}_i$ is the mean value of sequence $\mathbf{e}_i$, and $\widetilde{e}_i = \sqrt{\sum_{t=1}^{n} (E_{i,t}-\overline{e}_i)^2}$ is a measure of sequence $\mathbf{e}_i$ dispersion.

Input patterns $\mathbf{x}_i$ represent successive seasonal sequences which are centered and normalized. They have a zero mean, the same variance and unity length. Thus they are unified and differ only in shape. In contrast, patterns $\mathbf{y}_i$ are not globally unified and can express additional seasonality, e.g. when time series include both daily and weekly seasonalities, the former is expressed in the y-pattern shape, while the latter is expressed in y-pattern level and dispersion (compare x- and y-patterns in Fig. 2 and see discussion in \cite{Dud21a}). In the case of such seasonalities, we build forecasting models that learn from data representing the same days of the week, i.e. for test query pattern $\mathbf{x}$ representing Monday, the training set $\Phi=\{(\mathbf{x}_i,\mathbf{y}_i)\}_{i=1}^N$ consists of x-patterns representing all historical Mondays from the data and y-pattern representing the Tuesdays following them  (assuming $\tau = 1$).

The decoder based on the y-pattern forecasted by the network, $\widehat{\mathbf{y}}$, and coding variables describing the query sequence, $\widetilde{{e}}$ and $\overline{{e}}$, using transformed equation \eqref{eq1} for $\mathbf{y}$, calculates the forecasted seasonal sequence:

\begin{equation}\label{eq3}
\widehat{\mathbf{e}} = \widehat{\mathbf{y}}\widetilde{{e}}+\overline{{e}}
\end{equation}

Remarks:

\begin{enumerate}
    \item
RandNN was designed for forecasting time series with multiple seasonalities. In case of one seasonality, y-patterns express only one seasonality, and there is no need to decompose a forecasting problem. In case of no seasonality, the patterns express time series sequences of the same or different length. The input pattern length should be selected experimentally, while the y-pattern length is equal to the forecast horizon.

    \item
The bounds of the interval for random weights, $u$, correspond to the maximum sigmoid slope. To increase interpretability, let us express the bounds $u$ using the slope angles $\alpha_{max}$ \cite{Dud20}: $u=4 \tan \alpha_{max}$, and treat $\alpha_{max}$ as a hyperparameter. The second hyperparameter is the number of hidden nodes, $m$. Both hyperparameters decide about the bias-variance tradeoff of the model and should be tuned to the complexity of the target function.

    \item
Advantages of RandNN are extremly fast training and simplifying the forecasting task due to pattern representation.
In \cite{Dud21a}, it was shown that RandNN can compete with fully-trained NN in terms of forecasting accuracy, but is much faster to train.

\end{enumerate}

\section{Boosting of Ensemble Learning}

\subsection{Ensemble Learning Based on Residuals}

An ensemble based on residuals, which is a simplified variant of a gradient boosting algorithm \cite{Mas99}, is constructed sequentially. In the first step, a base model learns on the training set $\{(x_i,y_i)\}_{i=1}^N$ (we consider a scalar input and output for simplicity) and fits to original data. Let us denote this model $f_1(x)$ and the ensemble model including one member $F_1(x)=f_1(x)$. In the second step, the second base model is added, $f_2(x)$, such that the sum of this model with the previous one, $F_2(x)=F_1(x)+f_2(x)$, is the closest possible to the target $y$. In the successive steps, next base models are added with the same expectation. In the $k$-th step, the function fitted by the ensemble is $F_k(x)=F_{k-1}(x)+f_k(x)$. Thus, the mean squared error in this step, $MSE_k=\frac{1}{N}\sum_{i=1}^{N} \left(y_i-F_k(x_i)\right)^2$, can be written as:

\begin{equation}
MSE_k=\frac{1}{N}\sum_{i=1}^{N}\left(f_k(x_i)-r_{k-1,i}\right)^2
\label{MSE}
\end{equation}
where $r_{k-1,i} = y_i-F_{k-1}(x_i)$ is a residual between target and the response of the ensemble of $k-1$ members.

Equation \eqref{MSE} clearly shows that the base model added to the ensemble at the $k$-th stage fits to the residuals between the target and the ensemble built at stage $k-1$. So, it attempts to correct the errors of its predecessors. The training set for the base model in the $k$-th stage is $\{(x_i,r_{k-1,i})\}_{i=1}^N$. The final ensemble response is the sum of all member responses: $F_K(x)=\sum_{k=1}^{K} f_k(x)$.

Algorithm \ref{alg1} and Fig. \ref{FigEnsR} demonstrate a process of building the ensemble based on residuals (EnsR) for pattern-based forecasting. Learner 1 (RandNN) learns the original target function on $\Phi$. Each next learner learns the residuals between the target patterns $\textbf{y}$ and the aggregated outputs of its predecessors shown in the lower panel of Fig. \ref{FigEnsR}. Note that in EnsR, each learner can have different problem to solve, expressing different features. Learner 1 learns the specific patterns of seasonal cycles $\textbf{y}$, while the next learners learn completely different tasks, i.e. the residuals, which have not such distinct patterns as $\textbf{y}$ and have large stochastic component. This inconsistency between the problems solved by the learners can affect negatively the final result, especially in the common case of using identical base models (the same architecture and hyperparameters) at every stage of ensembling.

\begin{algorithm}[]
	\caption{EnsR: Boosting based on residuals}
	\label{alg1}
	\begin{algorithmic}

		\STATE {\bfseries Input:}\\

		\hspace{1mm} Base model $f$ (RandNN)\\
		\hspace{1mm} Training set $\Phi = \{(\mathbf{x}_i, \mathbf{y}_i)\}_{i=1}^N$\\
		\hspace{1mm} Ensemble size $K$\\

		\STATE {\bfseries Output:}\\
		\hspace{1mm} RandNN ensemble $F_K$\\
		\STATE {\bfseries Procedure:}\\
		\FOR{$k=1$ {\bfseries to} $K$}
		\STATE Learn $f_k$ based on $\Phi$ \\
		\STATE Calculate ensemble response $F_k(\mathbf{x}_i)=\sum_{l=1}^{k} f_l(\mathbf{x}_i), i = 1, ..., N$\\
        \STATE Determine residuals $\mathbf{r}_{k,i}=\mathbf{y}_i-F_k(\mathbf{x}_i), i = 1, ..., N$\\
        \STATE Modify training set $\Phi = \{(\mathbf{x}_i, \mathbf{r}_{k,i})\}_{i=1}^N$ \\
		\ENDFOR
	\end{algorithmic}
\end{algorithm}

\begin{figure}
	\centering
	\includegraphics[width=0.22\textwidth]{Res1.eps}
    \includegraphics[width=0.22\textwidth]{Res2.eps}
    \includegraphics[width=0.25\textwidth]{Res4.eps}
    \includegraphics[width=0.22\textwidth]{Res6.eps}
    \includegraphics[width=0.45\textwidth]{Res3.eps}
    \includegraphics[width=0.25\textwidth]{Res5.eps}
    \includegraphics[width=0.22\textwidth]{Res7.eps}

    \caption{EnsR: Responses produced by individual learners (upper panel) and the ensemble (lower panel). Target responses of individual learners in blue, target responses of the ensemble in black, real responses in red.}
    \label{FigEnsR}
\end{figure}

\subsection{Ensemble Learning Based on Corrected Targets}

To unify the problems solved by the learners at successive stages of ensembling, we modify the EnsR framework as follows. During sequential process, at stage $k$, the base model $f_k(x)$ is added to the ensemble. Ensemble response at this stage is an average of $k$ learners: $F_k(x)=\frac{1}{k}\sum_{l=1}^{k} f_l(x)$. The loss function can be expressed as:

\begin{equation}
\begin{align*}
MSE_k & =\frac{1}{N}\sum_{i=1}^{N}\left(y- \left(\frac{1}{k}\sum_{l=1}^{k-1} f_l(x) + \frac{f_k(x)}{k}\right)\right)^2\\
& =\frac{1}{Nk}\sum_{i=1}^{N}\left(f_k(x)-\left(ky-\sum_{l=1}^{k-1} f_l(x)\right)\right)^2
\end{align*}
\label{MSE2}
\end{equation}

As can be seen from this equation, the difference between $ky$ and the sum of the previous learners is the new target to which the $k$-th learner is fitted. This target expresses pattern $y$ corrected by aggregated residuals of the previous learners: $y+\sum_{l=1}^{k-1} (y-f_l(x))$. If the aggregated residuals are much smaller compared to the y-pattern (we expect this), the new targets at all stages of ensembling have similar shape of the y-pattern. So, all learners have similar tasks to solve and it is justified for them to have the same architecture and hyperparameters. This unburdens us from the awkward and time-consuming selection of the optimal model at each stage of ensembling.

Algorithm \ref{alg2} summarizes ensemble learning based on corrected targets (EnsCT) for pattern-based forecasting. Fig. \ref{FigEnsR2} shows the targets for learners at the successive stages, learners' outputs and the ensemble outputs. It was observed that at the initial stages of ensembling, the targets express the y-pattern shape, but this shape degrades gradually in the later stages - see the target for $K$-th learner in Fig. \ref{FigEnsR2}. Thus, the proposed EnsCT solves the problem of inconsistency between tasks learned at the successive stages of ensembling only partially.

\begin{algorithm}[]
	\caption{EnsCT: Boosting based on corrected targets}
	\label{alg2}
	\begin{algorithmic}

		\STATE {\bfseries Input:}\\

		\hspace{1mm} Base model $f$ (RandNN)\\
		\hspace{1mm} Training set $\Phi = \{(\mathbf{x}_i, \mathbf{y}_i)\}_{i=1}^N$\\
		\hspace{1mm} Ensemble size $K$\\

		\STATE {\bfseries Output:}\\
		\hspace{1mm} RandNN ensemble $F_K$\\
		\STATE {\bfseries Procedure:}\\
		\FOR{$k=1$ {\bfseries to} $K$}
		\STATE Learn $f_k$ based on $\Phi$ \\
		\STATE Calculate ensemble response $F_k(\mathbf{x}_i)=\frac{1}{k}\sum_{l=1}^{k} f_l(\mathbf{x}_i), i = 1, ..., N$\\
        \STATE Determine corrected targets $\mathbf{y}'_{k,i}=(k+1)\mathbf{y}_i-kF_k(\mathbf{x}_i), i = 1, ..., N$\\
        \STATE Modify training set $\Phi = \{(\mathbf{x}_i, \mathbf{y}'_{k,i})\}_{i=1}^N$ \\
		\ENDFOR
	\end{algorithmic}
\end{algorithm}

\begin{figure}
	\centering
	\includegraphics[width=0.22\textwidth]{ResC1.eps}
    \includegraphics[width=0.22\textwidth]{ResC2.eps}
    \includegraphics[width=0.25\textwidth]{ResC4.eps}
    \includegraphics[width=0.22\textwidth]{ResC6.eps}
    \includegraphics[width=0.45\textwidth]{ResC3.eps}
    \includegraphics[width=0.25\textwidth]{ResC5.eps}
    \includegraphics[width=0.22\textwidth]{ResC7.eps}

    \caption{EnsCT: Responses produced by individual learners (upper panel) and the ensemble (lower panel). Target responses of individual learners in blue, target responses of the ensemble in black, real responses in red.}
    \label{FigEnsR2}
\end{figure}

\subsection{Ensemble Learning Based on Opposed Response}

To prevent the degradation of the target shapes in the subsequent steps of ensembling, we propose ensemble learning based on opposed response (EnsOR).
The first step of the ensemble building procedure is the same as in EnsR and EnsCT.
At this stage, the base model $f_1(x)$ learns on original training set $\Phi$. An ensemble response is $F_1(x)=f_1(x)$.
The residual is calculated, $r_1 = y-F_1(x)$, and the "opposed" response pattern is determined as follows:

\begin{equation}
\hat{y}'_1=y+r_1=2y-F_1(x)
\label{Opp}
\end{equation}

The opposed response pattern expresses the target pattern with added the opposed error produced by the ensemble (see Fig. \ref{figOP}). The average of the response pattern $F_1(x)$ and the opposed response pattern $\hat{y}'_1$ gives the target pattern $y$. In the next step, base model $f_2(x)$ learns on the training set $\{(x_i,\hat{y}'_{1,i})\}_{i=1}^N$. Thus it learns the opposed response to reduce an ensemble residual. The ensemble response is calculated as an average of learners: $F_2(x)=\frac{f_1(x)+f_2(x)}{2}$. These operations are repeated in the following steps (see Algorithm \ref{alg3}). Namely, in step $k$, the opposed response pattern determined at stage $k-1$, $\hat{y}'_{k-1}=2y-F_{k-1}(x)$, becomes the target pattern for the learner $f_k(x)$. The ensemble response is an average of $k$ learners and the loss function at stage $k$ has the form:

\begin{equation}
MSE_k=\frac{1}{N}\sum_{i=1}^{N}\left(f_k(x_i)-\hat{y}'_{k-1,i}\right)^2
\label{MSE2}
\end{equation}
where $\hat{y}'_{k-1,i}=y_i+r_{k-1,i}=2y_i-F_{k-1}(x_i)$ is the opposed response pattern at stage $k-1$.

\begin{figure}
	\centering
	\includegraphics[width=0.6\textwidth]{Cor0.eps}
    \caption{Construction of the opposed response pattern.}
    \label{figOP}
\end{figure}

\begin{algorithm}[]
	\caption{EnsOR: Boosting based on opposed response}
	\label{alg3}
	\begin{algorithmic}

		\STATE {\bfseries Input:}\\

		\hspace{1mm} Base model $f$ (RandNN)\\
		\hspace{1mm} Training set $\Phi = \{(\mathbf{x}_i, \mathbf{y}_i)\}_{i=1}^N$\\
		\hspace{1mm} Ensemble size $K$\\

		\STATE {\bfseries Output:}\\
		\hspace{1mm} RandNN ensemble $F_K$\\
		\STATE {\bfseries Procedure:}\\
		\FOR{$k=1$ {\bfseries to} $K$}
		\STATE Learn $f_k$ based on $\Phi$ \\
		\STATE Calculate ensemble response $F_k(\mathbf{x}_i)=\frac{1}{k}\sum_{l=1}^{k} f_l(\mathbf{x}_i), i = 1, ..., N$\\
        \STATE Determine opposed response $\hat{\mathbf{y}}'_{k,i}=2\mathbf{y}_i-F_k(\mathbf{x}_i), i = 1, ..., N$\\
        \STATE Modify training set $\Phi = \{(\mathbf{x}_i, \hat{\mathbf{y}}'_{k,i})\}_{i=1}^N$ \\
		\ENDFOR
	\end{algorithmic}
\end{algorithm}

Fig. \ref{figEnsOR} shows learners' and ensemble responses in the following steps of ensembling. Dashed lines express the opposed responses which become targets for the based models in the next steps (blue lines). Note that the shape of the initial target pattern $\mathbf{y}$ keeps until the last stage. Thus, all learners learn on similar data and using identical base models at each stage of ensembling does not raise any objections, as in the case of EnsR.

\begin{figure}
	\centering
	\includegraphics[width=0.22\textwidth]{Cor1.eps}
    \includegraphics[width=0.22\textwidth]{Cor2.eps}
    \includegraphics[width=0.25\textwidth]{Cor4.eps}
    \includegraphics[width=0.22\textwidth]{Cor6.eps}
    \includegraphics[width=0.45\textwidth]{Cor3.eps}
    \includegraphics[width=0.25\textwidth]{Cor5.eps}
    \includegraphics[width=0.22\textwidth]{Cor7.eps}

    \caption{EnsOR: Responses produced by individual learners (upper panel) and the ensemble (lower panel). Target responses of individual learners in blue, target responses of the ensemble in black, real responses in red, and opposed responses in dashed red.}
    \label{figEnsOR}
\end{figure}

\section{Experimental Study}

In this section, we verify our proposed methods of ensemble boosting on four time series forecasting problems. They include short-term electrical load forecasting for four European countries: Poland (PL), Great Britain (GB), France (FR) and Germany (DE) (data was collected from \url{www.entsoe.eu}). The hourly load time series express three seasonalities: yearly, weekly and daily. The data period is four years, from 2012 to 2015. Atypical days such as public holidays were excluded from the data (between 10 and 20 days a year). The forecasting problem is to predict the load profile (24 hourly values) for each day of 2015 based on historical data. The forecast horizon is one day, $\tau=1$. The number of ensemble members was $K=50$. As a performance metric we use mean absolute percentage error (MAPE).

In the first experiment we evaluate an ensemble sensitivity to the base model hyperparameters: number of hidden nodes $m$ and size of the interval for hidden weights $\alpha_{max}$. Fig. \ref{figSens} shows the impact of hyperparameters on the test MAPE. Note the most sensitive to hyperparameters is EnsR while EnsOR is the least sensitive.
To quantitatively compare the sensitivities of ensemble variants, as a rough measure of sensitivity to a given hyperparameter, we define standard deviation of the test MAPE for the ensemble with optimal values of other hyperparameters:

\begin{equation}
S_m = Std(MAPE(\mathbf{y},F(\mathbf{x},\alpha^*_{max},m)))
\label{Sm}
\end{equation}
\begin{equation}
S_{\alpha_{max}} = Std(MAPE(\mathbf{y},F(\mathbf{x},\alpha_{max},m^*)))
\label{Sa}
\end{equation}
where the optimal hyperparameter values are marked with asterisks.

\begin{figure}
	\centering
	\includegraphics[width=0.7\textwidth]{Sens1.eps}
    \includegraphics[width=0.7\textwidth]{Sens2.eps}
    \includegraphics[width=0.7\textwidth]{Sens3.eps}
    \includegraphics[width=0.7\textwidth]{Sens4.eps}
    \caption{Ensemble sensitivity to RandNN hyperparameters.}
    \label{figSens}
\end{figure}

\begin{figure}
	\centering
	\includegraphics[width=0.24\textwidth]{PLk.eps}
    \includegraphics[width=0.24\textwidth]{GBk.eps}
    \includegraphics[width=0.24\textwidth]{FRk.eps}
    \includegraphics[width=0.24\textwidth]{DEk.eps}
    \caption{Ensemble error at successive stages.}
    \label{figk}
\end{figure}

From Fig. \ref{figSens} we can see that accuracy of EnsR and EnsCT deteriorates quickly with the number of hidden nodes and interval $U$ size, when these hyperparameters exceed their optimal values. This deterioration is related to the gradual loss of generalization for higher values of $m$ and $\alpha_{max}$. Deterioration for EnsOR is much slower, which means that this approach is more resistant to overtraining.

Table \ref{tab1} shows optimal hyperparameters, test and training errors and compares sensitivities of ensembling variants. The lowest values are in bold. Table  \ref{tab1} clearly shows the best performance of EnsOR. This method gave the lowest errors for each dataset and had the lowest sensitivity to both hyperparameters. By contrast, EnsR performed worst in terms of error and sensitivity. Note that the optimal hyperparameter values are smaller for EnsR and EnsCT than for EnsOR. This means that the base models in these boosting variants need to be less flexible (weaker) to prevent overfitting. Nevertheless, EnsR and EnsCT do not perform as well as EnsOR.

\begin{table}[htbp]

  \begin{center}
\setlength{\tabcolsep}{4pt}
  \caption{Optimal RandNN hyperparameters, errors and ensemble sensitivity.} \label{tab1}
    \begin{tabular}{llcccccc}
    \toprule
      Data & Ensemble & $m^*$ & $\alpha_{max}^*$ & $MAPE_{tst}$ & $MAPE_{trn}$ & $S_m$ & $S_{\alpha_{max}}$ \\
    \midrule
    \textbf{PL}
     & EnsR  & 20    & 40               & 1.18         & 0.76         & 0.641 & 0.436              \\
     & EnsCT & 20    & 60               & 1.15         & \textbf{0.72}         & 0.092 & 0.174              \\
     & EnsOR & 40    & 70               &\textbf{ 1.14}         &\textbf{ 0.72}         & \textbf{0.064} & \textbf{0.110}              \\
    \midrule
    \textbf{GB}
     & EnsR  & 20    & 10               & 2.55         & 1.55         & 0.509 & 1.531              \\
     & EnsCT & 10    & 50               & 2.52         & \textbf{1.51}         & \textbf{0.104} & 0.628              \\
     & EnsOR & 20    & 50               & \textbf{2.49}         & \textbf{1.51}         & \textbf{0.104} & \textbf{0.163}              \\
   \midrule
    \textbf{FR}
     & EnsR  & 10    & 10               & 1.65         & 1.27         & 0.466 & 0.853\\
     & EnsCT & 10    & 20               & 1.61         & 1.24         & 0.088 & 0.436\\
     & EnsOR & 20    & 40               & \textbf{1.57}         &\textbf{ 1.20}         & \textbf{0.058} & \textbf{0.069}\\
    \midrule
    \textbf{DE}
     & EnsR  & 10    & 40               & 1.29         & 1.37         & 0.774 & 0.572              \\
     & EnsCT & 10    & 70               & 1.21         & 1.26         & 0.138 & 0.141              \\
     & EnsOR & 40    & 80               & \textbf{1.17}         & \textbf{1.05}         & \textbf{0.036} & \textbf{0.063} \\
    \bottomrule
    \end{tabular}%
  \label{tab1}%
  \end{center}
\end{table}%

Fig. \ref{figk} demonstrates test MAPE in successive steps of ensembling. We can observe from this figure that for EnsOR the error converges faster with $k$ than for other ensemble variants. The convergence curve is also smoother for EnsOR than for its competitors. For them, in many cases, adding new member to the ensemble
causes a temporary error increase. Moreover, it was observed for EnsR that if the hyperparameters are too high (greater than optimal), which means a flexible base model, the error starts to successively increase with subsequent ensembling stages. This phenomenon is to a lesser degree observed for EnsCT, but not observed for EnsOR. Fig. \ref{figk2} demonstrates this problem for PL data and $m=100$, $\alpha_{max}=70$.

\begin{figure}
	\centering
	\includegraphics[width=0.5\textwidth]{PLk2.eps}
    \caption{Ensemble error at successive stages for too flexible base models.}
    \label{figk2}
\end{figure}

To improve further ensemble learning based on opposed response, we
introduce weights for training patterns which express their similarity to the query pattern. The more similar training pattern $\mathbf{x}_i$ to query pattern $\mathbf{x}$, the higher the weight. The similarity measure is a scalar product, $\mathbf{x}^T\mathbf{x}_i$. The opposed response takes the form:

\begin{equation}
\hat{\mathbf{y}}'_{k,i}=\mathbf{y}_i+w_i\mathbf{r}_{k,i},   i = 1, ..., N
\label{v1}
\end{equation}
where $\mathbf{r}_{k,i}=\mathbf{y}_i-F_k(\mathbf{x}_i)$ is a residual vector for the $i$-th training pattern and $w_i \in [0, 1]$ is the weight of this pattern.

In the base variant of EnsOR, the weights for all patterns are equal to one. In the typical ensemble learning, the weights are zero (the base models at each stage of ensembling learn on the original training set $\Phi$; such an approach we considered in \cite{Dud21}). Introducing weights, we balance between this two approaches.

As a weighting function, we consider four variants. In the simplest one, EnsOR1, we assume that the weighting function $g$ is just the scalar product:

\begin{equation}
g(\mathbf{x},\mathbf{x}_i) =\mathbf{x}^T\mathbf{x}_i
\label{sp}
\end{equation}

To avoid negative values of \eqref{sp} we can also use $g(\mathbf{x},\mathbf{x}_i) =\frac{1}{2}(\mathbf{x}^T\mathbf{x}_i+1)$ or replace negative values with zeros.

In the second variant, EnsOR2, we sort the training patterns according to similarity to the query pattern, from the most to the least similar. Let $r=1, ..., N$ be the rank of the training pattern in the similarity ranking. The weighting function expresses the linear dependence of the weight on the rank:

\begin{equation}
g(\mathbf{x},\mathbf{x}_i) =1+\frac{1-r_i}{N}
\label{rr}
\end{equation}

In the third variant, EnsOR3, the weighting function expresses the non-linear dependence of the weight on the rank:

\begin{equation}
g(\mathbf{x},\mathbf{x}_i) =\left(1+\frac{1-r_i}{N}\right)^d
\label{rr}
\end{equation}
where $d>1$ (we assume $d=4$).

In the fourth variant, EnsOR4, the most similar training patterns to the query pattern have unity weights, while the others have zero weights:

\begin{equation}
g(\mathbf{x},\mathbf{x}_i) =
\begin{cases}
    1       & \quad \text{if } \mathbf{x}_i \in \Xi_\kappa(\mathbf{x})\\
    0  & \quad \text{if } \mathbf{x}_i \notin \Xi_\kappa(\mathbf{x})
  \end{cases}
\label{nn}
\end{equation}
where $\Xi_\kappa(\mathbf{x})$ denotes a set of $\kappa$ nearest neighbors of query pattern $\mathbf{x}$ in $\Phi$.

An example of weights assigned to the training patterns by the above weighting functions are shown in Fig. \ref{figw}.

\begin{figure}
	\centering
	\includegraphics[width=0.5\textwidth]{Weights.eps}
    \caption{Examples of weights assigned to training pattern in different weighing variants.}
    \label{figw}
\end{figure}

Fig. \ref{fignn} allows us to evaluate the impact of the number of nearest neighbours $\kappa$ in EnsOR4 on the test MAPE. The optimal value of $\kappa$ depends on the data set: for PL $\kappa=14$, for GB $\kappa=40$, for FR $\kappa=38$, and for DE $\kappa=8$.

\begin{figure}
	\centering
	\includegraphics[width=0.24\textwidth]{PLnn.eps}
    \includegraphics[width=0.24\textwidth]{GBnn.eps}
    \includegraphics[width=0.24\textwidth]{FRnn.eps}
    \includegraphics[width=0.24\textwidth]{DEnn.eps}
    \caption{Ensemble error for EnsOR4 depending on $\kappa$.}
    \label{fignn}
\end{figure}

Table \ref{tab2} compares the proposed methods of pattern weighting. In this table results for individual RandNN and Ens1 are also shown. Ens1 is a classical (not boosted) ensemble of RandNN. Ens1 constructs the final forecast as an average of the responses of RandNN, which learn simultaneously on the same training set $\Phi$ \cite{Dud21}. As can be seen from this table, EnsOR4 outperformed its competitors as well as the classical ensembling Ens1. The much higher error for individual RandNN justifies fully ensembling. It is worth to mention that comparison of Ens1 with several forecasting models including statistical models (ARIMA, exponential smoothing, Prophet) and Machine learning models (MLP, SVM, ANFIS, LSTM, GRNN, nonparametric models) reported in \cite{Dud21} clearly showed that Ens1 outperforms all the competitive models in terms of accuracy.

\begin{table}[htbp]

  \begin{center}
\setlength{\tabcolsep}{4pt}
  \caption{Results for EnsOR with weighted patterns.} \label{tab1}
    \begin{tabular}{lcccccc}
    \toprule
      Data & EsnOR1 & EnsOR2 & EnsOR3 & EnsOR4 & RandNN $\cite{Dud21} & Ens1 $\cite{Dud21}  \\
    \midrule
    \textbf{PL}
     & 1.1419 & 1.1381 & 1.1329 & \textbf{1.1295} & 1.3206 & 1.1417  \\
    \textbf{GB}
     & 2.4935 & 2.4804 & 2.4822 & \textbf{2.4803} & 2.6126 & 2.5148  \\
    \textbf{FR}
     & 1.5668 & 1.5570 & 1.5524 & \textbf{1.5492} & 1.6711 & 1.5690  \\
    \textbf{DE}
     & 1.1743 & 1.1711 & 1.1683 &\textbf{1.1655} & 1.3809 & 1.1811  \\
    \bottomrule
    \end{tabular}%
  \label{tab2}%
  \end{center}
\end{table}%

\section{Conclusion}

Forecasting complex time series with multiple seasonalities is a challenging problem, which we solve using ensemble of randomized NNs. We propose three methods of boosting the RandNN ensemble. We showed than in the ensemble learning based on residuals, the base models have different tasks to solve at the successive stages of ensembling. Thus, the base model which is optimal at a given stage may not be optimal at other stages. To avoid selection of the optimal model at each stage of ensembling, which is unreasonable and too time consuming, we propose to unify the tasks solved at all stages. In such case, we can use the same base model (the same architecture and hyperparameters), RandNN in our case, which is optimal for all stages. To unify the tasks, we propose ensemble learning based on the corrected targets and ensemble learning based on the opposed response. The latter proved to be more resistant to task degradation at subsequent stages.

The experimental studies performed on four forecasting problems expressing triple seasonalities confirmed that the opposed response-based approach outperforms its competitors in terms of forecasting accuracy as well as sensitivity to both the base model hyperparameters and ensemble size. Further improvement of the winning solution was achieved by weighting the training patterns according to their similarity to the query pattern.

For further research, we plan to develop the methods of ensemble learning based on the opposed response for other types of learners, e.g. decision trees, and other types of problems (regression, classification).