\section*{Introduction}

\noindent Social activity of individuals follows certain rhythms at different time scales. Many of the individual activities, such as sending e-mails and making phone calls, are likely to be done in particular time intervals within a day (i.e., diurnal or circadian cycles), and the total daily activity could heavily depend on the day of the week (i.e., weekly cycles)~\cite{Malmgren2008PNAS,Malmgren2009Science,Jo2011PlosOne,Jo2012NewJPhys,aledavood2015digital,aledavood2015daily}. In general, these social activity rhythms emerging at different time scales may be correlated with each other; for instance, it has been shown that face-to-face contacts between classmates in a primary school follow a common diurnal cycle driven by the daily class schedule~\cite{Stehle2011PLOS,gauvin2014detecting}, yet at the same time they would also share activity rhythms at longer scales such as weekly and monthly, reflecting the annual school schedule.

  As social communications between humans form temporal social networks, financial transactions between banks also shape time-varying networks~\cite{May2008Nature,Haldane2011Nature,Barucca2016Soliton,Kobayashi2017arxiv,caccioli2018review}. In the interbank market, for instance, overnight bilateral lending and borrowing between banks organize temporal networks whose structure changes on a daily basis, because the overnight financial contracts last for only one day~\cite{Barucca2016Soliton,Kobayashi2017arxiv}. Thus, financial markets, similarly to social networks, can be interpreted as complex systems where each agent's activity should be captured and characterized at appropriate time scales~\cite{Barucca2016Soliton,Kobayashi2017arxiv}. In fact, many commonalities have recently been found between social communication patterns of humans and financial interaction patters of banks at particular temporal resolutions~\cite{Starnini2013PRL,Starnini2016SocNet,Kobayashi2017arxiv}.

 In recent years, \textit{non-negative tensor factorization} (NTF) has been frequently used to extract temporal activity patterns in various social contexts, such as face-to-face contacts~\cite{gauvin2014detecting,Sapienza2017arxiv}, Twitter posts~\cite{panisson2014mining} and players' matches in online games~\cite{sapienza2017NTFgame}.
 Among these studies, Gauvin et al.~\cite{gauvin2014detecting} showed that NTF is highly effective in detecting diurnal rhythms of students' activity in a primary school~\cite{Cattuto2010PLOS,Stehle2011PLOS}.
Characterizing diurnal rhythms lead the authors to uncover the multi-timescale community structure formed by classmates: students' temporal activity cycles, rather than the aggregated history of contacts, allows to reveal meaningful patterns that explain the complexity of children contact networks and communities.

Given the similarity between social and financial temporal dynamics, justified by underlying human factors, and the fact that multiple activity cycles are present at different frequencies in human social activities, our question is whether similar rhythms exist also in financial systems.
In this work, we uncover hidden multi-timescale patterns of banks' activities in the Italian online interbank market, e-MID~\cite{emidHP}. In previous studies, it has been recognized that there exist activity patterns in banks' financial transactions at a particular time scale, such as inter-day~\cite{Kobayashi2017arxiv} and intra-day scales~\cite{Iori2008JEDC,Beaupain2008,kobayashi2017significant}. However, it is still unknown whether these patterns coexisting at different time scales are dependent on each other, in which case banks exhibiting a given interday activity pattern are likely to follow a particular intraday pattern.

 We employ NTF as a tool to detect multi-timescale patterns, in which banks' activities are captured by a tensor having three dimensions: the list of banks, time of the day, and date.
 We will show that the PARAFAC decomposition of the 3-way tensor~\cite{Bro2003cc,kolda2009tensor} indicates that banks' trading activities can be classified into several patterns over the data period of 2001--2015. In particular, the NTF allows us to identify an anomalous pattern around the Lehman Brothers' collapse in September 2008. Our multi-timescale NTF approach reveals not only how banks facing the financial crisis changed their diurnal trading rhythms, but also on what dates such anomalies emerged.

 \new{Over the last couple of years, many proposals for measuring and controlling systemic risk have been presented, yet most of them focus on the static nature of financial networks~\cite{Beale2011,Gai2011,kobayashi2014efficient}. However, as we show in this study of the Italian e-MID interbank market, various patterns and anomalies  systematically emerge at different time scales, which should be taken into account in the design of financial regulations.}
 \new{Importantly, even in a situation in which bank activity has certain patterns at intra- and inter-day scales, it remains of paramount importance to understand whether the intra-day patterns are independent of the inter-day patterns (i.e., mono-timescale patterns) or they are correlated with each other (i.e., multi-timescale patterns).  }
 Our framework to extract multi-timescale patterns advances the understanding of when and how banks react to  systemic shocks, and in the future it could contribute to controlling the spread of financial systemic risk~\cite{May2008Nature,Haldane2011Nature,GaiKapadia2010,Brummitt2015PRE,Battiston2016PNAS}.

\section*{Data}
\label{data}
We use the time-stamped data on bilateral interbank transactions occurred between January 2001 and December 2015 in the Italian online interbank market (e-MID).
e-MID is an online platform  for financial institutions provided by the Italian company e-MID SIM S.p.A. based in Milan, Italy (henceforth, we call financial institutions as ``banks'' for brevity). Banks in need of lending or borrowing funds may find a counterpart by posting an order on the online platform, which makes e-MID a marketplace in which lenders and borrowers are matched.
Each transaction record in this data represents when a loan contract agreement has been reached between two banks and how much funds have been lent (in million Euros). In addition, the data contains information about the type of each loan agreement, i.e., a lender-proposed transaction or a borrower-proposed transaction. We define a lender-proposed (resp., borrower-proposed) transaction to be a transaction proposed by a bank that lends to (resp., borrows from) its counterpart.

As in the other interbank markets, the vast majority of transactions ($> 86$\%) in the e-MID market are overnight lending and borrowing (i.e., a loan contract lasts just for one day), while there are other types of transactions having longer maturity lengths such as two weeks, three month, etc.
To eliminate the influence of differences in maturity length, we focus on overnight transactions of unsecured Euro deposits labeled as ``ON'' (i.e., overnight) or ``ONL'' (i.e., overnight large, namely overnight transactions greater than 100 million Euros).
Table~\ref{tab:summary_stats} summarizes the basic statistics of this dataset.

 There are 3,839 business days over the data period between Jan 2, 2001 and Dec 31, 2015, on each of which transactions are made between 8:00 and 18:00. There are 289 banks that conducted at least one transaction over this period: 194 of them are Italian banks while the rest are foreign banks. \new{Regardless of the nationality of banks, we examine the transactions conducted between 8:00--18:00 in Italian standard time. We do not consider a time difference between countries where the headquarters of the banks are located.} The total number of transactions (ON and ONL) that we use in the analysis amounts to 1,148,699. It should be noted that the 289 banks are not always active over the data period (\textit{cf.,} Fig.~\ref{fig:numbanks}a). In fact, the daily average of the number of participating banks is around 95, and the average number of daily transactions is approximately 300 (\textit{cf.,} Fig.~\ref{fig:numbanks}b).
 \new{Although the time-series behaviors of the numbers of active banks and transactions are non-stationary, there is a stable scaling relationship between them (\textit{cf.,} Fig.~\ref{fig:numbanks}c)~\cite{Kobayashi2017arxiv}.}
 The dataset is commercially available from e-MID SIM S.p.A (\url{http://www.e-mid.it/}).

 \begin{figure}[t]
     \centering
     \includegraphics[width=.99\columnwidth]{numbanks_withscaling.pdf}
     \caption{Daily activities in the e-MID market. 20-day moving average is shown \new{for (a) number of active banks having at least one transaction on a day and (b) number of trades. (c) Scaling relationship between the numbers of active banks and trades. Each dot represents a day.} }
     \label{fig:numbanks}
 \end{figure}

\begin{table}[]
            \caption{Summary statistics of the Italian interbank market, e-MID.}
    \centering
    \begin{tabular}{lcccc}
    \hline
    \hline
   Data period & Jan 2, 2001 -- Dec 31, 2015 \\
   No.\ days & 3,839 \\
   Trading time & 8:00 -- 18:00 \\
   Maturity length & overnight (ON and ONL)\\
   No.\ banks (Italian banks) & 289 (194) \\
   No.\ total transactions  & 1,148,699 \\
   Average no.\ daily participants & 95.10 \\
   Average no.\ daily transactions & 299.22 \\
        \hline
    \end{tabular}\\
    \label{tab:summary_stats}
\end{table}

\section*{Method}

 As noted in the previous section, banks are not always active in the interbank market. At the daily scale, some banks have transactions only on a certain fraction of business days while having no transactions on the other days~\cite{Iori2008JEDC,Kobayashi2017arxiv}. If we look at intraday time scales, on the other hand, the frequency of a bank having at least one transaction is not uniformly distributed over the time intervals~\cite{Iori2008JEDC,Beaupain2008}. \new{These heterogeneities in temporal bank activity are illustrated in Fig.~\ref{fig:SI_hetero}.}

Since the maturity of loan contracts is overnight, it is important to identify what are the days in which banks are more likely to be active. Conversely, since the participation of banks in the trading activity can change over the course of a day, it is also crucial to understand intraday activity changes. Due to the multi-scale nature of bank trading activity, directly studying the volume of transactions of each bank over time would not allow us to detect inter- and intra-day dynamics. Thus, we need to disentangle each bank's activity across the two dimensions (i.e., time scales): interday and intraday. To this aim, we represent our data in a three-dimensional array (i.e., tensor), whose entries represent the amount of trades each bank performs at a given time of a given day. Given this multi-dimensional representation, our main goal is to extract some meaningful correlated multi-timescale activity patterns related to groups of banks sharing a similar amount of transactions at the same time (respectively, intraday and interday). This can be achieved by means of \textit{non-negative tensor factorization} (NTF), as described in the following.

\subsection*{Non-negative tensor factorization (NTF) at different time scales}\label{sec:NTF_description}

 Let us consider a three-dimensional tensor $\mathcal{X}\in \mathbb{R}^{I\times J\times K}$, where $I=N$ is the number of banks, $J=T$ is the number of time intervals of bank activity in a day (here between 8:00 and 18:00), and $K=D$ is the number of days. The entry $x_{ijk}\in \mathbb{Z}_{+}$ of the tensor $\mathcal{X}$ denotes the total amounts of trades (in million Euros) conducted by the $i$-th bank during $j$-th time interval of the $k$-th day.
 To extract the intra- and inter-day trading patterns characterizing banks with similar amount of transactions over time, we rely on the NTF, which decomposes the tensor $\mathcal{X}$ into the sum of several rank-one tensors, namely components \new{(\textit{cf.,} Fig.~\ref{fig:schematic}). } \new{For instance, if there exists a periodic inter-day pattern such as a two-day or a weekly cycle, then it would be captured by a component associated with the third dimension of the tensor.}

  \begin{figure}[t]
     \centering
     \includegraphics[width=.8\columnwidth]{schematic_NTF.pdf}
     \caption{\new{Schematic of the implementation of Non-negative Tensor Factorization (NTF) framework.}}
     \label{fig:schematic}
 \end{figure}

  Here, we implement the NTF by applying the  well-established method called PARAFAC (parallel factor analysis) or CANDECOMP (canonical decomposition) with non-negativity constraints~\cite{Bro2003cc,kolda2009tensor,lim2009nonnegative}. The decomposition can be written as
 \begin{align}
 \mathcal{X} \approx& \sum_{r=1}^{R}\mathbf{a}_{r}\circ \mathbf{b}_{r}\circ \mathbf{c}_{r},\label{eq:tensor} \\
 x_{ijk} \approx& \sum_{r=1}^{R}a_{ir}b_{jr}c_{kr},\label{eq:tensor_element}
 \end{align}
where $R\in\mathbb{N}$ denotes the number of components (i.e., the rank of the tensor), and the operator $\circ$ represents outer product.  Eq.~\eqref{eq:tensor} is the so-called canonical polyadic decomposition (CP) of  a tensor $\mathcal{X}$, where $\mathbf{a}_{r}\in \mathbb{R}_{+}^{N}$, $\mathbf{b}_{r}\in \mathbb{R}_{+}^{T}$ and $\mathbf{c}_{r}\in \mathbb{R}_{+}^{D}$ represent $r$-th component factors that respectively encode the membership of a bank to the component, the intervals in a day and the days in which the component is active (i.e. $\mathbf{b}_{r}$ and $\mathbf{c}_{r}$ are the intra- and inter-day activity patterns of the groups of banks in $\mathbf{a}_{r}$).

 Let $\mathbf{A}\in \mathbb{R}_{+}^{N\times R}$, $\mathbf{B}\in \mathbb{R}_{+}^{T\times R}$ and $\mathbf{C}\in \mathbb{R}_{+}^{D\times R}$ denote the factor matrices, whose $r$-th columns are the vectors $\mathbf{a}_{r}$, $\mathbf{b}_{r}$ and $\mathbf{c}_{r}$, respectively.  The factor matrices
$\mathbf{A}$, $\mathbf{B}$ and $\mathbf{C}$ are calculated by solving the following minimization problem with non-negativity constraints:
\begin{align}
 \min_{\mathbf{A}\geq 0,\mathbf{B}\geq 0, \mathbf{C}\geq 0} \lVert \mathcal{X} - \llbracket \mathbf{A},\mathbf{B},\mathbf{C}\rrbracket \rVert_{\rm F}^{2},
 \label{eq:parafac}
\end{align}
 where $\llbracket \mathbf{A},\mathbf{B},\mathbf{C}\rrbracket$ represents the Kruscal form of the tensor decomposition (i.e., the RHS of Eq.~\eqref{eq:tensor}), and $\lVert \cdot\rVert_{\rm F}$ denotes the Frobenius norm.

 This minimization problem can be rewritten to perform the minimization with respect to one factor matrix at a time. Let $\mathbf{X}_{(n)}$ denote a matrix created by the mode-$n$ matricization (or flattening) of tensor $\mathcal{X}$, where each row of $\mathbf{X}_{(n)}$ consists of a vector corresponding to a given index of mode-$n$. That is, $\mathbf{X}_{(1)},\mathbf{X}_{(2)}$ and $\mathbf{X}_{(3)}$ are $N\times TD$, $T\times ND$ and $D\times NT$ matrices, respectively. They can be expressed in terms of factor matrices as follows:
  \begin{align}
      \mathbf{X}_{(1)} &= \mathbf{A}(\mathbf{C}\odot\mathbf{B})^\top, \\
      \mathbf{X_{(2)}} &= \mathbf{B}(\mathbf{C}\odot\mathbf{A})^\top, \\
      \mathbf{X_{(3)}} &= \mathbf{C}(\mathbf{B}\odot\mathbf{A})^\top,
  \end{align}
  where $\odot$ denotes the Khatri--Rao product, which is the ``matching columnwise'' Kronecker product defined as $\mathbf{A}\odot\mathbf{B} = [\mathbf{a}_1\otimes\mathbf{b}_1,\ldots , \mathbf{a}_R\otimes\mathbf{b}_R]$. The minimization problem for PARAFAC decomposition Eq.~\eqref{eq:parafac} can be now reformulated as
  \begin{align}
      \left\{  \min_{\mathbf{A}\geq 0} \lVert \mathbf{X}_{(1)} - \mathbf{A}(\mathbf{C}\odot\mathbf{B})^\top \rVert_{\rm F}^{2}, \quad
        \min_{\mathbf{B}\geq 0} \lVert \mathbf{X}_{(2)} - \mathbf{B}(\mathbf{C}\odot\mathbf{A})^\top \rVert_{\rm F}^{2}, \quad
          \min_{\mathbf{C}\geq 0} \lVert \mathbf{X}_{(3)} - \mathbf{C}(\mathbf{B}\odot\mathbf{A})^\top \rVert_{\rm F}^{2},
      \right\}.
  \end{align}
  The minimizers of $\mathbf{A}$, $\mathbf{B}$ and $\mathbf{C}$ of this problem are computed using the non-negative alternate least squares method (ANLS) combined with the Block Principal Pivoting method (BPP) developed by \cite{kim2012fast}. Our implementation is based on the Tensor Toolbox~\cite{TTB_Software,TTB_Sparse} and the MATLAB codes  available from \cite{JinguKimHP}.

 \subsubsection*{Rank size}

To determine the number of components $R$ (i.e., rank size) used in the NTF model, we rely on the Core Consistency Diagnostic~\cite{Bro2003cc}.
  Given the tensor element $x_{ijk}$, we can rewrite Eq.~\eqref{eq:tensor_element} as
 \begin{align}
   x_{ijk} = \sum_{n=1}^{R}\sum_{m=1}^{R}\sum_{p=1}^{R}\lambda_{nmp}a_{in}b_{jm}c_{kp},
 \end{align}
 where $\lambda_{nmp}$ denotes the $(n,m,p)$ element of the superdiagonal binary tensor $\mathcal{L}$ (i.e., $\lambda_{nmp} = \delta_{nm}\delta_{mp}\delta_{np}$). Now let $g_{ijk}$ denote the $(i,j,k)$ element of the core tensor $\mathcal{G}$, which is obtained by fitting the data to the Tucker3 model~\cite{Bro2003cc}. In the Tucker3 model, the minimization problem reads
 \begin{align}
     \min_{\mathbf{G}} \lVert \mathbf{X} -\mathbf{A}\mathbf{G}(\mathbf{C}\otimes \mathbf{B})^{\top} \rVert_{\rm F}^{2},
 \end{align}
 where $\mathbf{X}$ is a $N\times TD$ matrix converted from tensor $\mathcal{X}$, and $\mathbf{G}$ is also a matricized version ($R_n\times R_{m}R_{p}$) of the core tensor $\mathcal{G}$. With the Tucker3 decomposition, the tensor is written as
 \begin{align}
     x_{ijk} = \sum_{n}^{R_n}\sum_{m}^{R_m}\sum_{p}^{R_p}g_{nmp}a_{in}b_{jm}c_{kp}.
 \end{align}
 From the lemma of Bro and Kiers~\cite{Bro2003cc}, the core tensor $\mathcal{G}$ will be identical to the superdiagonal tensor $\mathcal{L}$ if the Tucker3 model is perfectly fitted and the factors have full column rank. If $\mathcal{G}$ is significantly different from the superdiagonal tensor $\mathcal{L}$, by contrast, it means that there are non-negligible interactions between factors and the PARAFAC model is not appropriate. Thus, we could assess whether the data should be fitted to the PARAFAC model or the Tucker3 model by measuring the distance between $\mathcal{G}$ and $\mathcal{L}$.

 We employ the Core Consistency (CC) value proposed by Bro and Kier~\cite{Bro2003cc} as a measure of the distance  between $\mathcal{G}$ and $\mathcal{L}$:
 \begin{align}
     {\rm CC} = 100\times \left( 1-\frac{\sum_{n=1}^{R}\sum_{m=1}^{R}\sum_{p=1}^{R}(g_{nmp}-\lambda_{nmp})^2}{R} \right),
 \end{align}
 where we imposed the constraint $R_n = R_m=R_p = R$ in implementing the Tucker3 decomposition.
$\rm CC$ takes $100$ if the PARAFAC model perfectly fits the data and less than $100$ (possibly a negative value) if the model does not fit perfectly. Note that a rise in $R$ will reduce fitting errors while increasing the possibility of over-fitting. In general, as $R$ increases, there arises more interactions between components, which makes the core tensor $\mathcal{G}$ far from superdiagonal, resulting in a low value of $\rm CC$. Therefore, it is reasonable to stop increasing $R$ before the symptom of over-fitting emerges.
 Our determination of $R$ is given as
 \begin{align}
     R = \max R^\prime \in\{ R^\prime :\: {\rm CC}(R^{\prime})>L_{\rm cc},\: R^\prime\in \mathbb{N},  \},
     \label{eq:R}
 \end{align}
where ${\rm CC}(\cdot)$ denotes the core consistency as a function of the number of components, and $L_{\rm cc}$ is a threshold parameter. In line with \cite{Sapienza2017arxiv}, we set $L_{\rm cc} = 85$, and to minimize randomness created by a PARAFAC decomposition, we implement PARAFAC decomposition 20 times for a given rank size and use their mean as the value for ${\rm CC}(R^{\prime})$. Our implementation uses the MATLAB code available form \cite{corcondiaHP} which is based on the alternating nonnegativity-constrained least squares with block principal pivoting~\cite{papalexakis2015fastcorconia}.

\new{It should be noted that if one implements a low-dimensional clustering method, such as non-negative matrix factorization, separately at different time scales, then in general one would obtain different numbers of mono-timescale patterns at different scales. In that case, however, it would not be possible to know whether or not there exist multi-timescale patterns such that banks exhibiting a common intraday pattern also show a common interday pattern.}

 \section*{Results}

 \subsection*{Synthetic temporal financial markets}

  To validate the accuracy of our multi-timescale decomposition, we first implement NTF on a synthetic financial market whose properties are fully known \emph{ex ante}. Suppose that there are three groups of banks, each of which having particular intra- and inter-day trading patterns. First, we introduce \emph{intraday} trading patterns by considering the ``fitness'' of banks. In the fitness model, the probability that banks $i$ and $j$ trade at time $t$ is given by
  \begin{align}
  p_{ij,t}=a_{i,t}a_{j,t},
  \end{align}
  where $a_{i,t}\in[0,1]$ denotes the fitness of bank $i$ within the time interval $[t-\Delta, t]$~\cite{DeMasi2006PRE,Kobayashi2017arxiv}. $\Delta$ is the length of a time interval (in a day) expressed in minutes. Differences in the intraday trading patterns across groups are then represented by different values of fitness.
  Let us assume that the fitness evolves as illustrated in Fig.~\ref{fig:pattern_synthetic}a. Banks belonging to group 1 are most active in the early morning (blue solid), and banks in group 2 exhibit the highest trading activity around noon (red solid), and banks in group 3 are most active at the end of the day (black dotted). More specifically, the fitness value of a bank belonging to group $s$ at time interval $t$ is given as
  \begin{align}
    a_{i,t}^s = f(t;\mu_s,\sigma), \; t=1,\ldots,T,
  \end{align}
  where $f(t;\mu_s,\sigma)$ is the p.d.f. of normal distribution with mean $\mu_s$ and standard deviation $\sigma$. We set $(\mu_1,\mu_2,\mu_3) = (0,T/2,T)$ and $\sigma = T/4$.

   \begin{figure}[t]
     \centering
     \includegraphics[width=.85\columnwidth]{patterns_synthetic.pdf}
     \caption{Predefined intra- and interday trading patterns for the synthetic financial market. (a) Predefined intraday activity patterns, represented by the fluctuation of fitness. (b) Predefined interday activity patterns, given by the participation probability.}
     \label{fig:pattern_synthetic}
 \end{figure}

   Second, the \emph{interday} trading patterns are captured by variations in a bank's participation probability, which is the probability that a bank participates in the market on a given day, apart from whether or not the bank is able to find a trading partner \textit{ex post}. For day $d=1,\ldots,D$, the participation probability of bank $i$, denoted by $q_{i,d}$, is given as (Fig.~\ref{fig:pattern_synthetic}b)
   \begin{align}
   q_{i,d}=
       \begin{cases}
        0.5 & \text{if $i\in$ group 1,}\\
        \frac{2}{D}(d-1) & \text{if $i\in$ group 2 and $d\leq D/2$,}\\
        \frac{-2}{D}(d-D) & \text{if $i\in$ group 2 and $d>D/2$,}\\
        \frac{1}{D}(d-1) & \text{if $i\in$ group 3.}
       \end{cases}
   \end{align}
  For simplicity, we assume that trade volumes are identical for all trades so that the $(i,j,k)$ element of the 3-dimensional tensor is equal to the total number of trades that bank $i$ has during time interval $j$ on day $k$. \new{We will introduce a volume heterogeneity when analyzing the empirical data in the next section.}
  We set $N=120$, $T = 20$ (i.e., resolution $\Delta=30$ for hours between 8:00 and 18:00 in each day) and $D=1000$.

   \begin{figure}[t]
     \centering
     \includegraphics[width=.85\columnwidth]{activity_synthetic.pdf}
     \caption{PARAFAC decomposition for the synthetic financial markets. (a) Intra-day activity for each component. Activity in time interval $j$ of component $r$ is given by $b_{jr}$. (b) Inter-day activity for each component. Activity of day $k$ of component $r$ is given by $c_{kr}$.}
     \label{fig:activity_synthetic}
 \end{figure}
   \begin{figure}[t]
     \centering
     \includegraphics[width=.35\columnwidth]{CC_synthetic.pdf}
     \caption{Core-Consistency for the synthetic markets. Dashed line denotes $L_{\rm cc} = 85$.}
     \label{fig:CC_synthetic}
 \end{figure}

 Using the above synthetic tensor, we can now implement our multi-timescale tensor factorization. The factorization of tensor $\mathcal{X}$ allows us to measure the activity of each individual element: a bank, an intra-day time interval, and a day. The activity of intra-day time interval $j$ of component $r$ is given by $b_{jr}$, and analogously the activity of day $k$ of component $r$ is represented by $c_{kr}$.
 Fig.~\ref{fig:activity_synthetic} illustrates that PARAFAC decomposition well extracts the true multi-timescale patterns. The core-consistency value strongly suggests $R=3$ since CC(4) takes a large negative value.

\subsection*{Empirical data}

   Now we analyze empirical data. Given the accuracy of our method in the synthetic model, we expect the NTF method to extract latent multi-timescale patterns in the real-world financial market, e-MID. Here, we implement the $N\times T\times D$ tensor factorization described in Method, where we set intraday time resolution \new{(in minutes)} $\Delta =\{3,5,10,15,30,45,50\}$, and $D$ is set at the total number of business days between January 2001 and December 2015 (i.e., $D =$ 3,839). The number of active banks $N$, which traded at least once during the period, is 289. The Core-Consistency will reveal the number of components $R$, if any, which is equivalent to the the variety of existent activity patterns.

 \subsubsection*{Core-Consistency}

  \begin{figure}[t]
     \centering
     \includegraphics[width=.75\columnwidth]{CCvalue.pdf}
     \caption{Determination of rank size. (a) Core-Consistency values for $\Delta = 15$. Error bar denotes the 95\% confidence interval of the mean Core-Consistency value, calculated by 20 runs of PARAFAC decomposition. (b) Selected value of $R$ for a given time resolution $\Delta \in \{3,5,10,15,30,45,60\}$.}
     \label{fig:CC}
 \end{figure}

 The CC value for a given rank size is illustrated in Fig.~\ref{fig:CC}a for $\Delta =15$. The CC value becomes lower than 85 for $R\geq 4$, suggesting that $R=3$ should be selected. In general, imposing different intraday resolutions may lead us to choose different values of $R$ (Fig.~\ref{fig:CC}b). There is a tendency that a higher resolution requires a larger number of components for the PARAFAC decomposition to be appropriate. In the following, we employ $\Delta = 15$ and thereby $R=3$ as the benchmark case, the rationale of which will be discussed in the next subsection.

 \subsubsection*{Intra-day and inter-day activity}

   \begin{figure}[t]
     \centering
     \includegraphics[width=.9\columnwidth]{activity_resol15.pdf}
     \caption{Intra- and inter-day activities. Peak of the US house price (July 2006) and Lehman collapse (September 2008) are indicated by blue dotted and red dashed, respectively. The data for the US house price is taken from FRED (S\&P/Case-Shiller U.S. National Home Price Index)~\cite{Case-Shiller_FRED} (a) Intra-day activity. (b) Inter-day activity. (c) Share of a component in the daily activity. In panels (b) and (c), 20-day moving average is shown. }
     \label{fig:activity_resol15}
 \end{figure}

  Fig.~\ref{fig:activity_resol15}a shows \emph{intra-} and \emph{inter-day} activities for $\Delta = 15$, in which case there are three different patterns. The intra-day activity of Component 1 is characterized by its bimodal pattern.\footnote{Since the assignment of component index is originally arbitrary in the NTF implementation, we re-assign index in the ascending order of cumulative intra-day activity between 8:00 and 10:00.} It has two peaks around 10:00 and 17:00. On the other hand, both Components 2 and 3 have a single distinct peak in the early morning, after which their activities are very low. However, the distribution of intraday activity of Component 3 is more skewed than that of Component 2, which makes Components 1 and 3 two polar cases and Component 2 in between them. We would call the trading patterns of Components 1, 2 and 3 as \emph{normal trading}, \emph{early trading}, and \emph{flash trading}, respectively.

  The activity patterns for different temporal resolutions are also presented in Fig.~\ref{fig:SI_activity}. For the case of a finer intra-day resolution (e.g., $\Delta=5$, Fig.~\ref{fig:SI_activity}a), there are more than three components, and the activity of an additional component (i.e., Component 4) turns out to be very similar to the flash-trading pattern we saw in Fig.~\ref{fig:activity_resol15}a. On the other hand, when the temporal resolution is low (e.g., $\Delta=45$, Fig.~\ref{fig:SI_activity}c), there are only two components and the flash-trading pattern is no longer detected. Given these observations, it is natural to set $\Delta=15$ as a benchmark case, where apparently independent intra-day patterns can be captured. It should be noted that the following results are not sensitive to the temporal resolution level as long as $R=3$ is to be selected (e.g., $\Delta=30$, Fig.~\ref{fig:SI_activity}b).

 Fig.~\ref{fig:activity_resol15}b illustrates the \emph{inter-day} activity of each component. For Components 1 and 2, the activity decreased radically during the global financial crisis of 2007--2009, which was initiated by a significant decline of the US house prices (the peak date is indicated by blue dotted). In contrast, the activity of Component 3 spiked around the collapse of Lehman Brothers in September 2008 \new{(indicated by red dashed)}. Fig.~\ref{fig:activity_resol15}c shows how the share of a given component evolved over time.

   These observations suggests that Component 3 captures the ``crisis mode'' of bank trading while Components 1 and 2 represent more normal trading patterns. The crisis-mode interpretation is reinforced by the intra-day trading pattern of Component 3, in which banks trade only in the very early morning. In fact, we will show that, although we did not use information about the directionality of trades, this anomaly comes from the fact that banks in need of liquidity tried to obtain loans, rather than to extend credit, in the midst of the financial crisis.

\subsubsection*{Banks' affiliation to components}

\begin{figure}[t]
     \centering
     \includegraphics[width=.98\columnwidth]{memblevel_resol15.pdf}
     \caption{Affiliation of banks to components. The set of banks belonging to component $r$ is given by $I_r = \{i^\prime\}$ for which $a_{i^\prime r}$ is within the 90th percentile of $\{a_{ir}\}_{i=1}^{N}$. (a) Number of banks that belong to a component. (b) Jaccard index for the overlap of banks belonging to multiple components. (c) Average membership level on a given day. Membership level for a bank is given by a row of $\mathbf{a}_{r}\mathbf{c}_{r}^{\top}$. }
     \label{fig:membership_90}
 \end{figure}

 Each element of vector $\mathbf{a}_r$ represents the extent to which a bank's trading pattern is captured by Component $r$. Thus, we can extract a subgroup of banks whose trading patterns are characterized by Component $r$ at least to some extent.
 Here, the set of banks belonging to component $r$ is given by $I_r = \{i^\prime\}$ for which $a_{i^\prime r}$ is within the 90th percentile of $\{a_{ir}\}_{i=1}^{N}$ (Fig.~\ref{fig:membership_90}a). It should be noted that a bank may belong to multiple components if its trading pattern has multiple features (Fig.~\ref{fig:membership_90}b).

 The bank activity patterns emerging at the daily scale, captured by the row average of $\mathbf{a}_{r}\mathbf{c}_{r}^\top$, exhibit explicit differences across components (Fig.~\ref{fig:membership_90}c). While all types of banks have not been active since 2009, the normal-trading feature of banks belonging to Component 1 represents their behavior until 2008, and the trading pattern of banks belonging to Component 2 are dominated by the early-trading pattern up to 2006. On the other hand, the banks belonging to Component 3 exhibit daily patterns similar to the other types of banks prior to 2008, but suddenly their activity level spiked around the collapse of Lehman Brothers.

 \subsubsection*{Characterization of components}

  Aside from the intra- and inter-day trading patters, it would be of interest to see whether we may characterize each component by inherent attributes of the belonging banks.
  In the e-MID data set, a trading bank is classified into either of the following four types: aggressor lender, quoter lender, aggressor borrower and quoter borrower. In the e-MID online platform, a bank posts a request for loans or a proposal for lending, and a loan contract is made if another bank accepts the posted request. A bank is called ``aggressor lender'' when the bank accepts a request for loan posted on the e-MID platform, and the bank posted the request and borrowed fund is called ``quoter borrower''. ``quoter lender'' and ``aggressor borrower'' can also be understood analogously. In addition, we can also ask to what extent the nationality of banks can explain the differences in trading patterns.

 \begin{figure}[t]
     \centering
     \includegraphics[width=.9\columnwidth]{freqbar_resol15.pdf}
     \caption{Role of bank attributes in the characterization of components. (a) Average frequency of a bank being an aggressor lender, a quoter borrower, an aggressor borrower and a quoter lender. Error bar denotes 95\% confidence interval. (b) Share of Italian banks in each group. ``Random" denotes the average share of Italian banks under random chance, and error bar denotes the 90\% interval calculated from a binomial distribution with parameter $(N_r,p)$, where $N_r$ is the number of banks belonging to component $r$ and $p\equiv \# \text{Italian banks}/N$.
     }
     \label{fig:type_component_90}
 \end{figure}

 In fact, the majority of transactions in the e-MID market were done between aggressor lenders and quoter borrowers, which is a common property to all the component types (Fig.~\ref{fig:type_component_90}a). However, more than half of the banks belonging to Component 3 are quoter borrowers and the share of aggressor lenders is significantly lower than that. As for the nationality, the share of Italian banks among all the banks belonging to Component 1 is significantly low while the corresponding fraction is significantly large for the banks in Component 2 (Fig.~\ref{fig:type_component_90}b) at 90\% statistical significance, respectively. This suggests that foreign banks (i.e., banks from countries outside Italy) are likely to exhibit the normal-trading pattern, and Italian banks are more likely to employ the early-trading pattern than foreign banks do.

 These observations give us important implications about the multi-timescale trading patterns in the financial market.
 First, flash trading, which represents the intra-day trading pattern of Component 3, can be attributed mostly to quoter borrowers. Recall that flash trading was observed mostly in the period around the Lehman collapse in September 2008, but we do not know \textit{ex ante} whether such a trading pattern was driven either by demand or supply. The fact that Component 3 is largely attributed to quoter borrowers suggests that the flash-trading pattern during the financial crisis was conducted by banks that attempted to obtain liquidity as early as possible by posting quotes for loans. This may be regarded as evidence that some banks in fact faced a serious liquidity shortage at the time of the Lehman collapse~\cite{Brunnermeier2009JEP,Allen2010}. Our multi-timescale NTF approach reveals not only how banks reacted to the fear of liquidity shortage (i.e., intra-day pattern), but also specific dates on which the fear was most evident (i.e., inter-day pattern).

 Second, differences in the nationality of banks can lead to the variety of trading patterns. It turns out that foreign banks (Italian banks) are more likely to employ normal trading (early trading) than early trading (normal trading). On the other hand, the source of the crisis modality cannot be explained by the nationality of banks (Fig.~\ref{fig:type_component_90}b), suggesting that banks had similar chance of being in a crisis mode regardless of their nationalities.

\section*{Discussion}
 In this work, we presented an analytic framework based on  \textit{non-negative tensor factorization} (NTF) to extract temporal activity patterns of financial systems. Despite  previous studies on online financial markets recognized the existence of trading activity patterns  at specific time scales (e.g., inter-day~\cite{Kobayashi2017arxiv}, or intra-day~\cite{Iori2008JEDC,Beaupain2008,kobayashi2017significant}), we demonstrated---to the best of our knowledge for the first time---that activity patterns coexist at different time scales and depend upon each other. Our  methodology allowed us to uncover the hidden multi-timescale patterns of trading activities in an online interbank market (e-MID~\cite{emidHP}).

Within our framework,  banks' activities were represented by means of a tensor of three dimensions, namely the list of banks, time of the day, and date. Leveraging the power of NTF (based on the PARAFAC decomposition of the 3-way tensor~\cite{Bro2003cc,kolda2009tensor}), we uncovered the multi-timescale nature of financial trading dynamics in e-MID, suggesting that banks' trading patterns could be classified into subgroups exhibiting significantly different multi-timescale patterns. Our framework also allowed us to attribute roles to banking institutions (aggressor lender/borrower, or quoter lender/borrower) based on their trading patterns, yielding interpretable analytic insights.

By modeling the multi-timescale dynamics occurred over the period covered by our data (2001--2015), the proposed NTF-based framework allowed us to identify trading anomalies in the midst of the financial crisis. For example, around the time of Lehman Brothers' collapse in September 2008, our approach showed how banks changed their trading rhythms, and on what dates such anomalies emerged.

Understanding trading patterns in financial systems is of fundamental importance for many  reasons: first, a rigorous characterization of such systems' trading dynamics could help central banks to effectively intervene in the interbank market. In turn, this  would contribute to reducing systemic risk in the financial system as a whole.
Financial linkages created by bilateral transactions between banks can lead to a global network of interconnected risk, in which a failure of one bank can immediately spread over the entire system~\cite{May2008Nature,GaiKapadia2010,Brummitt2015PRE}.
Extracting multi-timescale patterns of financial markets allows for a better understanding of when and how banks react to  systemic shocks.

\new{Of course, the current approach has certain limitations. First, the tensor representation of interbank transactions does not capture information about the network aspect of the interbank market. While the current NTF takes bank ID, time and day as inputs, the interconnectivity between banks is still ignored. Given that many previous studies revealed that interbank networks contain rich information regarding systemic risk~\cite{May2008Nature,GaiKapadia2010,Brummitt2015PRE}, including structural information in the framework could improve the results.

Second, the current analysis did not fully explore the role of multi-scale activity patterns in propagating systemic risk. It would be interesting to examine to what extent each of the three activity patterns could promote or prevent financial contagion of bank defaults. To the best of our knowledge, the multi-timescale aspect of the source of financial systemic risk has not been investigated so far. This appears as a promising future direction of this line of research.

Finally, while the current work focused on the Italian e-MID market, this  is just a small part of the entire financial system. There are many other interbank markets in different countries, and there are also other types of interbank transactions such as trading of government bonds and credit default swaps. An important question to ask would therefore be: are the multi-timescale patterns found in our analysis ubiquitous or they are specific of markets that obey trading dynamics typical of the e-MID market?}

This work lays out the foundations for other researchers to study how  patterns and anomalies emerge in financial systems at different temporal resolutions:  in the future, such a framework could contribute to controlling the spread of financial systemic risk~\cite{May2008Nature,Haldane2011Nature,GaiKapadia2010,Brummitt2015PRE,Battiston2016PNAS}.
\new{It would be interesting to} investigate whether our method could be used both in predictive and prescriptive ways to help determine what could happen under certain risk conditions, as well as to provide possible  intervention strategies prior to shocks or cascading collapses in financial systems.


\section*{Abstract}
Echo state networks are powerful recurrent neural networks. However, they are often unstable and shaky, making the process of finding an good ESN for a specific dataset quite hard. Obtaining a superb accuracy by using the Echo State Network is a challenging task. We create, develop and implement a family of predictably optimal robust and stable ensemble of Echo State Networks via regularizing the
training and perturbing the input. Furthermore, several distributions of weights have been tried based on the shape to see if the shape of the distribution has the
impact for reducing the error.
We found ESN can track in short term for most dataset, but it collapses in the long run. Short-term tracking with large size reservoir enables ESN to perform strikingly with superior prediction. Based on this scenario, we go a further step to aggregate many of ESNs into an ensemble to lower the variance and stabilize the system by stochastic replications and bootstrapping of input data.

\section*{Introduction}
As a class of Reservoir Computing model, Echo State Networks accurately track and predict time series data by Machine Learning techniques. The realizations of
the reservoirs in real-world applications are various, such as object recognition, speech recognition, robotic movement control, dynamic pattern classification, and chaotic time-series generation etc. The main challenge ESN faces now is the stability. ESN is notoriously difficult to
track data because of the following reasons: the performance of the base learner is quite unsettling, which demands tremendous efforts for tweaking the parameter in the initial setting stage; the system would collapse when the dataset contains much noise; ESN can only track temporarily and it unable to forecast in the long run.

Since Echo State Networks first proposed by Jager (2007)\cite{jaeger_2007}, many modified models have been created to enhance the performance of the original base model in recent years. Kudithipudi et al.(2015)\cite{kudithipudi2015design} designed a toroidal ESN architecture with hybrid topology to lower power consumption. Ma et al. (2016)\cite{ma_shen_chen_wang_wei_yu_2016} proposed Functional Echo State Networks (FESN) to enhance the separability of different classes in a high-dimensional functional space. Mcdermott and Wikle (2017)\cite{mcdermott_wikle_2017} constructed Quadratic Echo State Networks for ensemble building so that the machine can handle uncertainty in nonlinear spatio-temporal forecasting.

The main contribution of this paper is to state the challenges of Echo State Networks and provide potential remedies to improve the performance of the system by building ensemble of ESNs for robustly stable and highly predictive machine. We create, develop and implement a family of predictably optimal robust and stable
ensemble of Echo State Networks via regularizing the training and perturbing the input. Furthermore, constructing several distributions of weights based on the
shape to see if the shape of the distribution has the impact for reducing the error. ESN can track in short term for most dataset, but it collapses in the long run. Short-term tracking with large size reservoir enables ESN to perform strikingly with superior prediction. Based on this scenario, we go a further step to aggregate many of ESNs into an ensemble to lower the variance and stabilize the system. We create the ensemble of Echo State Networks with two scenarios: (i) Adaptation on structured pattern matrices (ii) Stochastic replications and bootstrapping of input data.

The remainder of this paper is organized as follows. Section 2 gives brief introduction of Echo State Networks and then discusses the challenges we encounter to obtain superb accuracy in prediction from mainly three aspects: randomness in weights; distribution of weights; environmental and parameter setting. Section 3 and 4 elaborate the challenges demonstrated in the previous chapter and provide experimental results based on different datasets. Section 3 focus on the exploration of weight distribution. Section 4 investigates environmental setting the for optimal performance. Section 5 illustrates the tremendous improvement in prediction achieved by Ensembles of Echo State Network. Section 6 provides some final conclusions and directions for future work.

\section{Echo State Networks}
Traditional Neural Networks provide a large family of function approximators able to handling the situations where the underlying true function is heavily nonlinear. It is done by statistically modeling in a way that attempts to mimic the human brain. Recurrent Neural Networks go further step to allow the system handle time dependent data due to the distributed hidden state and non-linear dynamics. The distributed hidden state can efficiently store large past information and the non-linear dynamics are able to update the hidden state in sophisticated ways. \cite{sutskever_hinton_2010}

Proposed by Jager (2007)\cite{jaeger_2007}, Echo State Networks as one of Recurrent Neural Networks (RNNs) hold the idea of merely learning the output layer with early layers random and fixed. Typical Echo State Networks embody three layers: an input layer, a hidden layer and an output layer. Consider a typical ESN with K, N, L number of units in its input layer, hidden layer and output layer accordingly. The input units at time step n is denoted as $\mathbf{u}(n) = [u_1(n),u_2(n),...,u_K(n)]$, the units in the hidden layer is $\mathbf{x}(n) = [x_1(n),x_2(n),...,x_N(n)]$, and $\mathbf{y}(n) = [y_1(n),y_2(n),...,y_L(n)]$ is the units in the output layer. The input weights $\mathbf{W}^{in}$ with the form of $N\times K$ matrix connect the input units and the hidden units, and the internal weights $\mathbf{W}$ with the form of $N\times N$ matrix connect the internal units,
both are randomly generated. Only the output weights $\mathbf{W}^{out}$ with the form of $L\times (K+N+L)$ matrix connect the internal units and the output units are trained. The Figure 1 below showed all possible connections of Echo State Networks, including the feedback from $y(n-1)$ to $\hat{x}(n)$, the connection straight from the input to the output, and also the connection between output units.
\begin{figure}[!htbp]
\begin{center}
\includegraphics[scale=0.5]{0.png}
\end{center}
\caption{Typical Echo State Networks With All Possible Connections}
\end{figure}
For training typical ESNs, two steps are contained: first is to have the states updated from the previous step, second is to train the output weight $\mathbf{W}^{out}$:
\begin{align}
\mathbf{x}(n) &= \mathbf{f}(\mathbf{W}^{in}\mathbf{u}(n)+\mathbf{W}\mathbf{x}(n-1) + \mathbf{W}^{back}\mathbf{y}(n-1))\\
\mathbf{y}(n) &= \mathbf{f}^{out}(\mathbf{W}^{out}(\mathbf{u}(n), \mathbf{x}(n), \mathbf{y}(n-1)  ))
\end{align}
where $\mathbf{f} = (f_1,f_2,...,f_N)$ are the activation functions of hidden units (usually sigmoid function, Hyperbolic Tangent or ReLU function in ESN) and $\mathbf{f}^{out} = (f^{out}_1, f^{out}_2,..., f^{out}_L)$ are the activation function of output units (usually linear function). Only the output layer weights need to be trained with simple linear regression algorithms or others learning algorithm. Thus echo state network models have simplified training algorithms compared to other recurrent neural networks.

\section{Challenges of ESN}
Traditional recurrent neural network can process a sequence of inputs dependent on each other but may encounter the issue of vanishing gradient problem and these algorithms are suffering from memory storage requirement training time restriction. To remedy these problems, echo state network can be used as a partially-trained recurrent neural network. It can handle the issues by having a relatively large reservoir with sparsely connected neurons. The connections in the reservoir are randomly assigned and the weights in the input layer and the reservoir layer are not trained. The advantage of this approach is in the incredibly simple training procedure since most of the weights are assigned only once and at random. However, the simplicity in training also bring challenges in robustness. Randomness assigned in the reservoir creates instability and yields high variance. In addition, the base learner demands complicated tuning in parameter setting, which requires tremendous efforts. Some questions come up along with challenges:
What kind of randomness we can try? Does uniform random weight or Gaussian random weight matter? What’s the range of random number? Shall we use fixed leaking rate or dynamic one? In this section we mainly explore two aspects of challenges: (I) Randomness in weights (II) Parameter turning efforts for optimal environment.

\subsection{Randomness in Weights}
The randomness in weights brings potential instability and creates high variance in the machines. Echo State Networks (ESN) are particularly sought-after because of the inherent structure not needing heavily computational back propagation. Traditional Neural Networks use back propagation with expensive computation process, while Echo State Networks avoid this painful step.  Unfortunately, ESN replaces back propagation with random assignments or values to reservoir oscillators.
\begin{figure}[H]
\begin{center}
\includegraphics[scale=0.25]{1.png}
\end{center}
\caption{Random Assignment In Simple ESN}
\end{figure}

What sort of distribution to be used in random assignments? To explore the randomness of the weights, several distributions are worthy being tried. Here based on the shape we explored four kinds of distributions: Uniform distribution, Gaussian distribution with the same variance as the uniform distribution, Gaussian distribution with the same range as the uniform distribution, Arcsine distribution. All of them have significant difference, it is interesting to see if the shape of the distribution has the impact for reducing the error.
\begin{figure}[H]
\begin{center}
\includegraphics[scale=0.25]{2.png}
\end{center}
\caption{Probability Density Function of Four Different Distributions}
\end{figure}

\subsection{Preprocessing for Proper Scaling}
Similar to other neural networks in environment setting, there are too many parameters and hyper-parameters to be tuned and adjusted. There are several ways for data preprocessing and parameter tuning:
\begin{itemize}
\item Data preprocessing\\
Several ways can be used to scale the dataset as preprocessing methods. As far as preprocessing on the input space is concerned, wherever deemed necessary, we may consider the following for each entry $x_{ij}$  of $\mathbf{X}$:
\begin{itemize}
\item Cubitize:
$$ x_{ij} \leftarrow \frac{x_{ij}-min(\mathbf{x}_j)}{max(\mathbf{x}_j)-min(\mathbf{x}_j)} $$
\item Standardize:
$$ x_{ij} \leftarrow \frac{x_{ij}-mean(\mathbf{x}_j)}{\sqrt{variance(\mathbf{x}_j)}} $$
\item Unitize::
$$ x_{ij} \leftarrow \frac{x_{ij}-mean(\mathbf{x}_j)}{\|\mathbf{x}_j-mean(\mathbf{x}_j)\mathbf{l}_p \|_2} $$
$$\textup{where }\|x_j-mean(\mathbf{x}_j)\mathbf{l}_p\|_2 = \sqrt{\sum_{i=1}^n[x_{ij}-mean(\mathbf{x}_j)]^2} $$
\end{itemize}
\item Dynamic leaking rate\\
Leaking rate $\alpha$ in Echo State Networks in the rang of $(0,1]$ is considered as the speed of reservoir update dynamics in continuous time \cite{lukosevicius_2012}. The choice for leaking rate effects the performance dramatically. In addition, the status of leaking rates as fixed or dynamic can also bring different results. The update process is:
\begin{align}
\tilde{x}[n] &= tanh(W_{in}u[n]+Wx[n-1])\\
x[n]&=(1-\alpha)x[n-1] + \alpha\tilde{x}[n]\\
y[n] &= f^{out}(W_{out}(u[n], x[n]))
\end{align}
when $\alpha = 1$, it's the special case as $x[n]=\tilde{x}[n]$ the model does not have leaky integration.
\item Perturbation on each time step\\
The idea of adding random perturbation, also known as white noise, to the the input at each time step comes from perturbation theory, a general mathematical methods. We add a small random perturbation to the input as they will become the initial conditions for the next time step integration. After adding perturbation $\tau$ the update process is:
\begin{align}
\tilde{x}[n] &= tanh(W_{in}u[n]+Wx[n-1] + \tau)\\
x[n]&=(1-\alpha)x[n-1] + \alpha\tilde{x}[n]\\
y[n] &= f^{out}(W_{out}(u[n], x[n]))
\end{align}
\item Perturbation on initialization\\
In the initial step, the value of $x[0]$ is set as 0 by default. But now we give it small perturbation with different distributions to see if the small variability can make any difference:
\begin{verbatim}
  x0 = rep(0,resSize)           # original value
  x0 = rnorm(resSize, 0, 0.1)   # perturbation scenario 1
  x0 = runif(resSize,0,1)       # perturbation scenario 2
\end{verbatim}
So the update process is:
\begin{align}
x[0] &= x_0,\qquad x_0 \neq 0\\
\tilde{x}[n] &= tanh(W_{in}u[n]+Wx[n-1])\\
x[n]&=(1-\alpha)x[n-1] + \alpha\tilde{x}[n]\\
y[n] &= f^{out}(W_{out}(u[n], x[n]))
\end{align}
\end{itemize}

\section{Effect of Weight Distribution}
As mentioned in the previous chapter, we have explored four different kinds of distributions of weights based on shapes.
The most common distribution many paper used is Uniform distribution. Gaussian distributed weights is another popular ones used in Echo State Network. Here we take two elements into account for the comparison with the Uniform distributed weights. We try to control the range of the distribution or the variance of the distribution, to see the which kind of Gaussian distributed weights have better predictive capability. Arcsine distribution is noticeable for its U shape. Based on the original dataset from time-series data Mackey-Glass (delay=17) by Mantas Lukosevicius \cite{lukosevicius_2012}, we explore four different scenarios.
Dataset Mackey-Glass time series refers to the following delayed differential equation:
\begin{align}
\frac{dx}{dt}=\beta x(t)+\frac{\alpha x(t-\tau)}{1+x(x-\tau)^{10}}
\end{align}
where x is the series in the time t, and $\tau$ is the time delay. In this case time delay is 17.
\begin{figure}[H]
  \begin{center}
  \includegraphics[width=4in]{3.png}
  \caption{Sample of Mackey-Glass (delay=17) Data}
  \end{center}
\end{figure}

\subsection{Uniform Distributed Weights}
The initial attempt for the option of distributed weights is uniform distribution due to its simple and interpretable form. Typically the error measure used here is mean-absolute-error (MAE) or mean-square-error (MSE) or root-mean-square-error (RMSE) showed below, and we use MSE for in this paper.
\begin{align}
E_{MAE}(\mathbf{y},\mathbf{y}^{target}) &= \frac{1}{T}\sum_{i=1}^T|y_i-y_i^{target}|\\
E_{MSE}(\mathbf{y},\mathbf{y}^{target}) &= \frac{1}{T}\sum_{i=1}^T(y_i-y_i^{target})^2\\
E_{RMSE}(\mathbf{y},\mathbf{y}^{target}) &= \sqrt{\frac{1}{T}\sum_{i=1}^T(y_i-y_i^{target})^2}
\end{align}
So here for Uniform distributed weights (in both input layer and hidden layer). From the plot below shows the predictive capability of the machine for the target signal performs very well.
\begin{figure}[H]
  \begin{center}
  \includegraphics[width=4in]{4.png}\\
  \caption{Target Signal Tracking for Uniform Distributed Weights}
  \end{center}
\end{figure}

\subsection{Gaussian Distributed Weights}
Two elements are taken into account for the comparison between the Uniform distributed weights and Gaussian distributed weights. We control the shape (range) of the distribution or the variance of the distribution to see which kind of Gaussian distributed weights generate better performance.

\subsubsection{Gaussian Distributed Weights Same Variance}
To control the variance, it means the weights of uniform distribution and weights of Gaussian distribution have the same variance. In this case, the variance is $\sigma^2=1/12$. So the weights follow normal distribution $\mathcal{N}(0, \frac{1}{12})$.
It gives a better result than the uniform one. The MSE for this case is $9.581 \times 10^{-6}$ while the uniform one is $2.099 \times 10^{-5}$
And from the signal tracking plot Fig. 6., it shows this machine can predict the signal quite well.
\begin{figure}[H]
  \begin{center}
  \includegraphics[width=4in]{6.png}\\
  \caption{Signal Tracking: Gaussian Distributed Weights with Same Variance}
  \end{center}
\end{figure}

\subsubsection{Gaussian Distributed Weights Same Range}
For Gaussian distribution with the control of the range,  we try to keep the weights of uniform distribution and weights of Gaussian distribution in almost the same range $[-0.5,0.5]$. Because for Gaussian Distribution $x\in (-\infty, +\infty)$ is in the open range, we set 99.7\% of the data within the range $[-0.5,0.5]$ , which means points that fall more than 3 standard deviations from the norm are likely outliers. So $3\sigma= 0.5, \sigma^2=1/36$.
\begin{figure}[H]
  \begin{center}
  \includegraphics[width=4in]{7.png}\\
  \caption{ Three-Sigma Rule of Thumb }
  \end{center}
\end{figure}

And we get MSE for Gaussian distributed weights machine $0.0219 $. MSE is much larger than the previous two, showing this kind of weights has poor performance. And from the signal tracking plot Fig. 8., it obviously reflects that the signal can be tracked at the beginning but collapse in the middle part. So this distribution cannot track the signal very well. It may be useful in short-term tracking, but cannot be used in the long run.
\begin{figure}[H]
  \begin{center}
  \includegraphics[width=4in]{8.png}\\
  \caption{Signal Tracking: Gaussian Distributed Weighted with Same Range}
  \end{center}
\end{figure}

\subsection{Arcsine Distributed Weights}
\newcommand{\vw}{\mathbf{w}}
To explore the distribution of weights with different shapes, Arcsine distribution cannot be ignored as it has an absolutely distinct shape than other distributions.
Technically, a continuous random variable $W$ is said to have an arcsine distribution on $\left[-\frac{1}{2}, +\frac{1}{2}\right]$, if its probability density function is given by (13), it can also be easily established that the CDF for $W$ is given by (14)
\begin{align}
 f_W(\vw)&=\frac{2}{\pi\sqrt{(1-2\vw)(1+2\vw)}}\\
  F_W(\vw)&= \frac{2}{\pi} \texttt{arcsine}\Bigg(\sqrt{\frac{2\vw+1}{2}}\Bigg)
\end{align}
In fact, in general, one can define arcsine random variables in $[a, a+l]$, with pdf as (15), and the corresponding CDF for $W$ given by (16)
\begin{align}
     f_W(\vw)&=\frac{1}{\pi\sqrt{(-a+\vw)(a+l-\vw)}}\\
    F_W(\vw) &= \frac{2}{\pi} \texttt{arcsine}\Bigg(\sqrt{\frac{-a+\vw}{l}}\Bigg)
\end{align}
Probability density function of Arcsine distribution is a symmetric U-shaped curve, centered on $(x_{max} - x_{min})/2$, with highest close to the two extrema, and quite flat over the central region. So the shape for Arcsine distribution is quite unique. In this case we get MSE of weights with the distribution that follows arcsine(-0.5, 0.5). MSE here is $ 1.746 \times 10^{-6}$ even smaller than the one from Gaussian distribution with the same variance, showing Arcsine distributed weights have edge over other distributed weights. And from the signal tracking plot Fig. 9., the signal can be tracked super well. So for the time-series data Mackey-Glass (delay=17), weights with Arcsine distribution perform the best.
\begin{figure}[H]
  \begin{center}
  \includegraphics[width=4in]{9.png}\\
  \caption{Signal Tracking: Arcsine Distributed Weighted}
  \end{center}
\end{figure}

\subsection{Simulation Based on Four Weight Distributions}
For further explore the properties of different distributions of the weights, we generate various time series dataset:
\begin{itemize}
\item Non-stationary time series data without trend
\begin{align}
 X_t = \epsilon_t + 0.81X_{t-1}+0.72\epsilon_{t-1}
\end{align}
\item Non-stationary time series data with trend
\begin{align}
 X_t = \epsilon_t + 0.81X_{t-1}+0.72\epsilon_{t-1} + \frac{t}{1000} + \Big(\frac{t}{1000}\Big)^2
\end{align}
\item Stationary time series data without trend
\begin{align}
 X_t = \texttt{sin}((1+t)\pi^3)
\end{align}
\item Stationary time series data with trend
\begin{align}
 X_t =\texttt{sin}((1+t)\pi^3) + \frac{t}{1000}
\end{align}
\end{itemize}
\begin{figure}[H]
  \begin{center}
  \includegraphics[width=6in]{10.png}\\
  \caption{Different Time Series Datasets}
  \end{center}
\end{figure}

After intense exploration on four different scenarios we found that for non-stationary time-series data, none of the distribution can handle them and all of them explode. For stationary time-series data without trend, Uniform distributed weight gives the best performance, and other distributed weights generate pretty good prediction as well. For original time-series data and stationary time-series data with trend, Arcsine distributed weight can better track the signal with smaller error. In the table below we denote original Mackey Glass (delay=17) data as MG, stationary time series data without trend as SWT, stationary time series data with trend as ST, Gaussian-A means Gaussian distribution with same variance and Gaussian-B means Gaussian distribution with same range.
\begin{table}[H] \centering
  \caption{Result of Exploration on Weight Distribution}
\begin{tabular}{@{\extracolsep{5pt}} ccccc}
\\[-1.8ex]\hline
\hline \\[-1.8ex]
 MSE & Uniform & Gaussian-A & Gaussian-B& Arcsine\\
\hline \\[-1.8ex]
 MG & 2.099$\times 10^{-5}$ & 9.581$\times 10^{-6}$ & 0.0219  & 1.746$\times 10^{-6}$ \\
  SWT &   $1.804 \times 10^{-6}$  & $ 5.479 \times 10^{-6}$ & $2.237 \times 10^{-5} $ & 3.100$\times 10^{-5}$ \\
  ST & $2.221 \times 10^{-5}$  & $9.030 \times 10^{-6}$ & 0.00439 &  $1.253 \times 10^{-6}$\\
\hline \\[-1.8ex]
\end{tabular}
\end{table}

\iffalse
\begin{table}[H]
\centering
\begin{threeparttable}
\caption{Exploration Result}
\begin{tabular}{ |c|c|c|c|c| }
\hline
 MSE & Uniform & Gaussian-A & Gaussian-B& Arcsine\\ \hline\hline
 MG & 2.099$\times 10^{-5}$ & 9.581$\times 10^{-6}$ & 0.0219  & 1.746$\times 10^{-6}$ \\ [0.5ex]
 \hline
  SWT &   $1.804 \times 10^{-6}$  & $ 5.479 \times 10^{-6}$ & $2.237 \times 10^{-5} $ & 3.100$\times 10^{-5}$ \\
 \hline
  ST & $2.221 \times 10^{-5}$  & $9.030 \times 10^{-6}$ & 0.00439 &  $1.253 \times 10^{-6}$\\
 \hline
\end{tabular}

\begin{tablenotes}
 \scriptsize
\item[1] \textbf{MG = original Mackey Glass (delay=17) data}
\item[2] \textbf{SWT = stationary time series data without trend}
\item[3] \textbf{ST = stationary time series data with trend}
\item[4] \textbf{Gaussian-A = Gaussian distribution with same variance}
\item[5] \textbf{Gaussian-B = Gaussian distribution with same range}
\end{tablenotes}
\end{threeparttable}
\end{table}
\fi

From the table above we notice that the shape of the distribution indeed has impact on the accuracy of the tracking. Particularly, the arcsine distribution appears to offer a better result probably due to induce some level of sparsity, which in turn reduces the error.

\section{Tuning Exploration}
As mentioned in second chapter about the environmental setting challenge for Echo State Network, here we will continue and elaborate this topic to different kinds of time series datasets to see how the environmental settings change according to different time series datasets, and how to select the optimal strategy based on the performance from different datasets.

\subsection{Performance}
After exploring different time series datasets with different traits listed above, we discover that Echo State Networks can give decent performance on the majority of stationary dataset without surprise. Additionally, it can also track the information of some non-stationary datasets in short-term. But the machine breaks down in the long run for most non-stationary datasets. Table 2 shows the comparative predictive performance on various time series datasets. To thoroughly explore the performance of ESNs, we also compare ESNs with other common methods for time series prediction such as feed-forward neural networks (NN) and autoregressive integrated moving average model (ARIMA). Based on different dataset for ESN method, we first adjust the datasets for the optimal setting via data preprocessing and parameter turning.
\begin{figure}[H]
  \begin{center}
  \includegraphics[width=2.5in]{13.png}
  \includegraphics[width=2.5in]{14.png}
  \caption{Good Performance of ESN Tracking}
  \end{center}
\end{figure}
\begin{table}[H] \centering
  \caption{Comparative Predictive Performance on Relative Datasets}
\begin{tabular}{@{\extracolsep{5pt}} ccccc}
\\[-1.8ex]\hline
\hline \\[-1.8ex]
 Dataset  & Tuning Setting & $MSE_{ESN}$ & $MSE_{NN}$ & $MSE_{ARIMA}$ \\
\hline \\[-1.8ex]
 lynx & \makecell{ cubitize\\ dynamic leaking rate\\pertubation on initialization}  & 0.0848 &0.0492 &  0.0615 \\ [0.5ex]
 \hline
  gas  & \makecell{ cubitize\\ dynamic leaking rate\\pertubation on each time step}& 0.0035 & 0.0006 & 0.0008 \\
 \hline
  sunspots  & \makecell{ scaling\\ dynamic leaking rate\\pertubation on initialization}& 3.9231 & 0.0103 & 0.0173  \\
 \hline
  AirPassengers & \makecell{ cubitize\\ dynamic leaking rate\\pertubation on initialization\\pertubation on each time step}& 0.0062 & 0.0415  & 0.1037 \\
 \hline
 wineind  & \makecell{ cubitize\\ dynamic leaking rate\\pertubation on each time step}& 0.0794& 0.0682 & 0.0458 \\
 \hline
 taylor  & cubitize & 0.0011  & 0.1760 & 0.0756 \\
 \hline
 gold  & \makecell{ cubitize\\ dynamic leaking rate\\pertubation on initialization\\pertubation on each time step}&   0.0099 & 0.0078 & 0.096 \\
 \hline
 woolyrnq  & \makecell{ cubitize\\ dynamic leaking rate}& 0.1672 & 0.0205 &  0.04671 \\

\hline \\[-1.8ex]
\end{tabular}
\end{table}
For feed-forward neural network and AIRMA method, we haven't extended our work on their environmental setting so the result is not their optimal performance. However, the three methods show the results almost in the same magnitude, which indicates that ESN generates reasonable results as a type of neural network. Furthermore, more efforts are desired for generating seemly performance. Efficient learning environment setting includes:
data preprocessing via cubitization and scaling; perturbation on initialization; perturbation for weights; dynamic leaking rate etc.
In summary, the system collapses when the dataset contains too much noise. The size of the dataset matters and ESN tends to give poor performance on small dataset for lack of information. The machine can track in short term for most dataset, while it cannot handle long-term non-stationary time series dataset. Furthermore, based on different dataset, different methods would be employed. Some need more randomness (perturbation effect, modification on
initialization etc.), while some need to lower the variation (ensemble building, dynamic leaking rate etc.)

\section{Ensemble of Echo State Networks}
Based on the above exploration on the performance from different datasets, we can see that even under the ideal scenarios, the randomness still yields high variance on estimators. The challenge we encounter drive us to look for the potential for Ensembles such as Bagging \cite{breiman_1996} and Random Forest \cite{breiman_last_rice_2000} as an approach to reduce variance and thus generate better results. Bagging, Boosting, Random Forest have in common their inherent capacity to achieve Bias-Variance trade-off and hence have a reduction in prediction error. Especially in big data situations, selecting a single model cannot provide the optimal prediction. Except random forest and boosting in some degree, ensemble methods do not give variable selection or variable importance measure. The reason for this is that the motivation of ensemble learning is predictive optimality instead of selection of a single model.

The basic idea of Ensemble Echo State Networks is that based on the given data $\mathcal{D} = \{(\mathbf{x_i}^T,\mathbf{y_i})^T, i=1,...,n\}$, we want to find out a true but unknown function $f$ that models the functional relationship between variable $X$ and $Y$. Here the function $f$ represents Echo State Networks $f^{(esn)}$. Assume we are given $M$ different estimators of $f^{(esn)}$:
\begin{align}
\mathcal{E} = \{\hat{f}_1, \hat{f}_2, ... , \hat{f}_M   \}
\end{align}
and assume that each estimator $\hat{f}_m(\cdot)$ has a corresponding relative importance measured by weights $\alpha_m$ for $m=1,...,M$. So the goal is to find an estimator of $f$ with the smallest prediction error. That is to say, ensemble learning is to combine all the available decent candidate estimators, and $  \hat{f}(\cdot)$ is a weighted average of base learners:
\begin{align}
\hat{f}^{(ENS)}(\cdot) = \sum_{m=1}^M \alpha_m \hat{f}_m(\cdot)
\end{align}
Each of the estimator  $\hat{f}_m(\cdot)$ is referred to as a base learner. If the base estimators  $\hat{f}_m(\cdot), m =1,...,M$ are strongly correlated, the performance of the ensemble will probably decrease due to the redundant base learner results. Hence, it is significant to select the base learners and decide the weights based on their performance. Another point worthy of note is that the ensemble estimator $\hat{f}^{(ENS)}(\cdot)$ ought to be at least as good as the best of the base learners:
\begin{align}
\overline{err}( \hat{f}^{(ENS)}(\cdot))\leq \underset{m=1,...,M}{argmin}\{\overline{err}(\hat{f}_m(\cdot) )\}
\end{align}
Therefore, how to build each $\hat{f}_m(\cdot)$ and how to find $\hat{\alpha_m}$ are the key points for good ensembles. There are two kinds of base learners: homogeneous base learners and heterogeneous base learners depending on if all base learners come from the same function class. For heterogeneous base learners, one can combine different kinds of base learners such as SVM classifiers and trees. For homogeneous base learners, all base learners are from the same class such as all from linear models with each has a different subset of variables trained on different random subsets of the same data, or all from trees built on random subsets of the same data as it is the case in tree bagging and random forest.
Regardless of how the base learners are obtained, the resulting aggregation is equation (26).
Here we use homogeneous base learners to build ensembles because the base learners functions are all from Echo State Networks. Here we mainly explode two scenarios for building ensemble base learners: Perturbation of Weights and Bootstrap Aggregating.

\subsection{Perturbation of Weights}
First scenario is perturbation of the input weights $\mathbf{W}^{in}$ and the weights in hidden layer $\mathbf{W}$. If $\mathbf{W}^{in}$ and $\mathbf{W}$ change at each step, then the corresponding $\hat{f}_m(\cdot)$ changes. So many copies of $\hat{f}_m(\cdot)$ can be generated by changing weights $W^{in}$ and $W$. The step for this scenario:
\begin{itemize}
\item Generate $W^{in}_m$ and $W_m$
\item Form $\hat{f}_{m}(\cdot)$ for every perturbation of weights
\item Build ensemble on the above $\hat{f}_m(\cdot)$
$$ \mathcal{E}(W^{in}, W) = \{ \hat{f}_m(\cdot), m=1,2,...,M, W_m^{in}, W_m  \}$$
\end{itemize}

\begin{algorithm}[H]
\caption{Perturbation on Weights}
\begin{algorithmic}
\FOR{$m \gets 1:M$}
\STATE Generate $W^{in}$ and $W$ with fixed dataset
\STATE Form $\hat{f}_{m}(\cdot)$ based on different $W_m^{in}$ and $W_m$
\ENDFOR
\STATE Build ensemble: $\hat{f}^{ENS}(u^{new}) = \{\hat{f}_m(u^{new}), m=1,2,...,M \}$
\end{algorithmic}
\end{algorithm}

\subsection{Bootstrap Aggregating}
Bootstrap aggregating is one of machine learning ensemble algorithm designed to improve the stability and accuracy in statistical classification and regression. It also reduces variance and helps to avoid overfitting. It is usually used in decision tree methods, but it can be applied with any type of method. Bootstrap aggregating was proposed by Leo Breiman in 1994\cite{breiman_1994}. His motivation was to achieve variance reduction in classification and regression, and the aggregation of many replicates of an estimator is more stable than a single instance of that estimator.

Given dataset $\mathcal{D} = \{(x_i,y_i), i=1,2,...,n \}$ along with $M$ estimators $\hat{f}_{m}(\cdot), m = 1,2,...,M $ of $f: \mathcal{X} \rightarrow  \mathcal{Y}$, instead of choosing $1$ out of the $M$, we aggregate all the $M$ $\hat{f}_{m}(\cdot)$. To get $\hat{f}_{m}(\cdot)$, we first use bootstrap to create many copies of the data $\mathcal{D}$, and then for each copy we create $\hat{f}_{m}(\cdot)$. The step for this scenario:
\begin{itemize}
\item Uniformly sample the given training dataset $\mathcal{D}$ of size $n$ with replacement
\item Generate M new training datasets $\mathcal{D}_m$ of size $n'$
\item When $n = n' $ for large n, $\mathcal{D}_m$ has the fraction $ (1 - 1/e)$ of the unique examples of $\mathcal{D}$, and duplicates the rest \cite{Aslam}
\item For each new data $\mathcal{D}_m$ create $\hat{f}_{m}(\cdot)$
\item Build ensemble on the above $\hat{f}_{m}(\cdot)$
\end{itemize}

\begin{algorithm}[H]
\caption{Bootstrap Aggregating (Time Series Dataset)}
\begin{algorithmic}
\STATE Initialize $m:=0$
\REPEAT
\STATE $ m := m+1$
\STATE Form a bootstrap sample $\mathcal{D}_{m} = \{(x_1^{(m)}, y_1^{(m)}),...,(x_n^{(m)}, y_n^{(m)})\}$ by randomly sampling with replacement n times from $\mathcal{D} = \{(x_1, y_1),...,(x_n, y_n)\}$
\STATE  Arrange the data in each new dataset $\mathcal{D}_{m}$ in time order
\STATE  Construct the bootstrapped estimator $\hat{f}_{m}(\cdot)$ based on $\mathcal{D}_{m}$
\UNTIL{$m=M$}
\STATE Compute the bagged estimator $\hat{f}^{ENS}(u^{new}) = \frac{1}{M}\sum_{m=1}^M\hat{f}_{m}(u^{new})$
\end{algorithmic}
\end{algorithm}
The ensemble estimator $\hat{f}^{ESN}(\cdot)$ is driven by the hyper-parameter $M$, the number of base learners (base echo state networks) in the ensemble. A good way to determine the optimal value of $M$ for a given problem is tuning using cross validation
\begin{align}
M^{(opt)} = \underset{M\in grid(M) }{\texttt{argmin}}\{\texttt{CV} ( \hat{f}^{ESN} )\}
\end{align}
As M increases, $\texttt{Variance}(\hat{f}^{ESN})$ decreases without any increase in $\texttt{Bias}(\hat{f}^{ESN})$ which leads to bias-variance trade-off and optimal predictive performance as a result.

\subsection{Simulations}
For the eight time series datasets we explored in the fourth chapter above, we extend the exploration and build Ensembles of Echo State Networks for these datasets with the approaches of randomization on weights. We found that the Ensembles consistently outperform the base learners. The mean square error of base learner is denoted by $\bar{\mathcal{E}}_{s}$ and the mean square error of ensemble estimator is $\bar{\mathcal{E}}_{ens}$ in the table below.

\begin{table}[H] \centering
  \caption{MSE Comparison and Error Reduction}
\begin{tabular}{@{\extracolsep{5pt}} cccc}
\\[-1.8ex]\hline
\hline \\[-1.8ex]
 Dataset & $\bar{\mathcal{E}}_{s}$ & $\bar{\mathcal{E}}_{ens}$ & Error Reduction  \\
\hline \\[-1.8ex]
 lynx & 0.0848  &     0.0661    &  22.05\%   \\
  gas &  0.0035   &    0.0016     &   54.28\%  \\
  sunspots &  3.9231  & 0.5120  & 86.95\%       \\
  AirPassengers & 0.0062   &   0.0058     &  6.45\% \\
 wineind &   0.0794     &   0.0635    &  20.03\% \\
 taylor &   0.0011       &  0.0005 &     54.55\%\\
 gold &  0.0099   &    0.0057     & 42.42\%\\
 woolyrnq & 0.1672    &     0.1373   &    17.88\% \\
\hline \\[-1.8ex]
\end{tabular}
\end{table}

The errors drop significantly with the ensemble approach under the optimal base learner scenario we achieved in the previous chapter. As the input weights and the weights in the hidden layer are selected at random, the model is going to perform differently given diverse weights. The prediction result for single ESN inevitably contains uncertainty. Therefore, ensemble ESN with the reasonable number of qualified members can extract the inherent feature through uncertainty and also give credit to relative weak learners. Here we regard each base learner equally weighted.

For ensemble with bootstrap method, the pivot idea is to draw sample from training set (subsets of training dataset). So the dataset itself could not be too small. Here we use large dataset Mackey-Glass (delay=17) time series data from Chapter 3, containing 10000 value. We get:
\begin{align*}
\bar{\mathcal{E}}_{s} &= 2.099\times10^{-5}\\
\bar{\mathcal{E}}_{ens} &= 7.865\times10^{-6}\\
 \Delta\bar{\mathcal{E}}&= \left|\frac{7.865\times10^{-6} - 2.099\times10^{-5}}{2.099\times10^{-5}}\right| = 62.53\%
\end{align*}
The result illustrates that the Ensemble of Echo State Networks can improve the performance and reduce the variation caused by randomness if the base learners provide reasonable result. The stability of the model is enhanced by ensemble technique.

\section{Conclusion and Future Work}
Echo state networks are compelling recurrent neural networks with the sparse random connections in the hidden layer reflecting the previous states. Nevertheless, the process of obtaining a decent ESN for particular datasets is pretty tough due to its instability. ESN is notoriously difficult to track data because of the following reasons:
\begin{itemize}
\item The performance of the base learner is quite unsettling, which demands tremendous efforts for tweaking the parameter in the initial setting stage.
\item The system would collapse when the dataset contains much noise.
\item ESN can only track temporarily and it unable to forecast in the long run.
\end{itemize}
In order to conquer this challenging we create, develop and implement a family of predictably optimal robust and stable ensemble of Echo State Networks by regularizing the training and perturbing the input. In addition, several distributions of weights with significant features have been studied to see if the shape of the distribution has the effect for reducing the error. We notice that the shape of the distribution indeed has impact on the accuracy of the tracking. Particularly, the arcsine distribution appears to generate better results due to the level of sparsity, which in turn reduces the error.

During the exploration on the environmental setting we conclude that the efficient learning environment setting includes: data preprocessing via cubitization and scaling; perturbation on initialization; perturbation for weights; dynamic leaking rate. Furthermore, ESN can track in short term for most cases, but it collapses in the long run. Short-term tracking with large size reservoir enables ESN to perform strikingly with superior prediction.

Based on this scenario, we go a further step to aggregate many of ESNs into an ensemble to lower the variance and stabilize the system. We create the ensemble of Echo State Networks with two scenarios: (i) Adaptation on structured pattern matrices (ii) Stochastic replications and bootstrapping of input data. Ensembles of ESN turn out to give pretty good performance. One thing worthy of note is that the fundamental base learner should be at least give reasonable result so that the ensemble can work and achieve magnificent prediction. In the most basic ensembles, all the base learners are equally weighted, the same as the work in this paper. However, it is not necessary the best way. Finding optimal weights for ensemble model is actually one of the most common challenges. Future work can explore this and also more work needs to be done to build stable base ESNs, so that ensemble model can do well.

\nocite{*}

\nolinenumbers