Co-evolutionary multi-task learning for dynamic time series prediction
Rohitash Chandra Yew-Soon Ong Chi-Keong Goh



 


 

 label1, label2 ]Rohitash Chandra 
 label3]Yew-Soon Ong 
label3]Chi-Keong Goh 
 


[label1]  Centre for Translational Data Science, The University of 
Sydney, Sydney, NSW 2006, Australia 



[label2]  School of Geosciences, The University of 
Sydney, Sydney, NSW 2006, Australia 


[label3] Rolls Royce @NTU Corp Lab,  Nanyang Technological 
University, 42 Nanyang View, Singapore  
 


 

 


  

Time series prediction typically 
consists of a data reconstruction phase where the time series is broken into 
overlapping windows known as the timespan. The size of the timespan  can be 
seen as a way of determining the extent of past information required for   an 
effective prediction.    
In certain applications such as the prediction of wind-intensity of storms and cyclones,  prediction 
models need to be dynamic in accommodating different values of the timespan. 
These applications require robust prediction  as soon as the event takes place.  
We identify a new 
category of problem called   dynamic time series 
prediction that requires a  model  to give prediction when 
presented with varying lengths  of the timespan.    In this paper, we 
propose a  
co-evolutionary multi-task learning method that provides a synergy between 
multi-task learning and co-evolutionary algorithms to address dynamic time 
series prediction. The method features effective use of building blocks  of 
knowledge   inspired by dynamic programming and multi-task learning. It 
enables 
neural 
networks to   retain 
modularity  during training for  making a decision in situations even when 
certain inputs are missing.    The effectiveness of 
the method
is demonstrated using  one-step-ahead chaotic time series and tropical cyclone 
wind-intensity prediction.   




 Coevolution; multi-task learning; modular neural networks;   chaotic
time series; and  dynamic programming.









§ INTRODUCTION

 

 

 
Time series 
prediction  typically involves 
a pre-processing stage where the original time 
series is reconstructed into a state-space representation that is used as 
dataset for training models such as neural networks 
<cit.>. The reconstruction involves 
breaking 
the time series using overlapping  windows known as  
timespan   taken 
at regular intervals which defines the time lag   <cit.>. The 
optimal values for  
 timespan and time lag are needed for effective 
prediction. These values   vary 
on the type of problem and require  costly 
computational evaluation for model selection; hence, some effort has  been 
made to address this issue. Multi-objective and competitive coevolution 
methods have been used to take advantage of different features from the 
timespan    during training 
<cit.>. Moreover, neural network have been used  
for determining optimal timespan of selected  time series 
problems <cit.>.

In  time series  for natural 
disasters such as cyclones 
<cit.>, it is 
important to develop models that can make predictions dynamically, i.e. the 
model  has the ability to make a prediction as soon as any 
observation or data is 
available. The minimal value for the timespan can have huge impact 
for the case of cyclones, where data is only  available every 6 hours   
<cit.>. A way to address such categories  of problems is 
to devise 
robust training algorithms and models that are capable of performing given 
different types of input or subtasks. 
We define  dynamic 
time 
series prediction as a problem that requires dynamic  
prediction given a set of input features that vary in size. It 
has been highlighted in recent work <cit.>  that recurrent 
neural networks trained with a predefined timespan   can only 
generalise well for the 
same timespan  which makes   dynamic time series 
prediction a challenging problem. 
Time series prediction problems can be generally characterised into three major 
types of problems 
that include   one-step   
<cit.>, multi-step-ahead  
<cit.>, and multi-variate time 
series prediction <cit.>. These problems 
at times may overlap with each other, for instance, a multi-step-ahead 
prediction can have a multi-variate component. Similarly, a one-step prediction 
can also have a multi-variate component, or a one-step ahead prediction can be 
used for multi-step prediction and vice-versa.  In this paper, we  
identify  a special class of problems that require dynamic prediction 
with the hope that the trained model can be useful for different instances of 
the problem.   
 

 
Multi-task learning employs  shared representation knowledge for learning  
multiple 
instances from  the same problem  with the goal to develop   models with 
improved   performance in decision making
<cit.>.
 We note that different values in the timespan can be 
used to generate several distinct datasets that have overlapping features which 
can  be  used to train modules for shared knowledge representation as needed 
for multi-task learning. Hence, it is important to ensure that modularity is 
retained   in such a way so that decision making can take place 
even when certain inputs are missing. Modular neural networks have been 
motivated from repeating 
structures in nature   and applied  for visual recognition 
tasks   <cit.>.   
Neuroevolution   has been used to optimise  performance and connection 
costs  in modular neural networks <cit.> which also has  the 
potential of  
learning 
new tasks without forgetting old ones <cit.>. The  features of 
modular learning provide motivation to be incorporated with multi-task learning 
for dynamic time series prediction. 
 
 
  In dynamic programming, a  large problem is broken down into 
 sub-problems, from which at least one sub-problem is used as a  building 
block  for the  optimisation problem. Although dynamic programming has been 
primarily used for optimisation problems, it  
 has been briefly   explored for data driven  
learning  <cit.> <cit.>. The notion of using 
sub-problems as building block in dynamic programming can be used in   
developing algorithms for  multi-task learning. Cooperative coevolution (CC) 
is a divide and conquer approach that divides a problem into subcomponents that 
are implemented as sub-populations <cit.>. CC has been effective 
  for   learning difficult  problems using neural networks <cit.>. 
Potter and De Jong  demonstrated that CC  provides more 
diverse solutions through the sub-populations when compared to conventional 
evolutionary algorithms <cit.>. CC has been very effective for 
training recurrent neural networks for time series prediction    problems 
<cit.>. 
 
  Although multi-task learning has mainly been used for machine learning 
problems, the concept of shared knowledge  representation has motivated other 
domains.  In the 
optimisation literature, 
multi-task evolutionary algorithms have been proposed  for   exploring and  
exploiting    
common knowledge 
between the tasks and enabling transfer of knowledge between them for 
optimisation <cit.>. It was demonstrated that 
knowledge 
from related  tasks can help in speeding up the optimisation process and 
obtain 
better  quality solutions when compared to conventional (single-task 
optimisation)  
approaches.   Evolutionary multi-task learning has 
been used for efficiently training feedforward neural networks for   n-bit 
parity problem <cit.>, where different subtasks were 
implemented as  different topologies  that obtained  improved training 
performance. In the literature, synergy 
of  dynamic programming, 
multi-task learning and neuroevolution has not been  explored.  Ensemble 
learning methods would be able to address 
dynamic time series  to an extent,  where an ensemble is defined by the 
 timespan of 
the time series. Howsoever, it would not have the feature of shared knowledge 
representation that is provided through multi-task learning. Moreover, there is 
a need for a  unified  model for dynamic times series problems  due to problems 
 that require dynamic prediction. 
 
In this paper, we propose a  
co-evolutionary multi-tasking method that provides a synergy between 
multi-task learning, dynamic programming and coevolutionary algorithms. The 
method   enables neural networks to 
be trained by featuring shared and modular knowledge representation  in order 
to make predictions given limited input features. 
This enables the learning process to employ modules of knowledge from  the 
related subtasks as building blocks of knowledge for a  unified  model. The 
proposed method 
is used for one-step-ahead chaotic time series problems using feedforward 
neural 
networks for  benchmark problems.    The  method 
is also  used for  
 tropical cyclone 
wind-intensity prediction and addresses the problem of minimal timespan where 
 dynamic prediction is required.  The paper extends results presented 
 in <cit.>.
  

The rest of the paper is organised as follows. Section 2 gives a background on 
multi-task learning,  
cooperative neuro-evolution, and time series prediction. Section 3 gives 
details of the co-evolutionary multi-task learning method for dynamic time 
series prediction. 
Section 4 presents the results with discussion and Section 5 presents the  
conclusions and directions for future research.

  
 


§ BACKGROUND AND RELATED WORK

 



 §.§ Multi-task learning and applications



 A number of approaches have been presented that 
considers multi-task learning  <cit.> for different types of 
problems that include 
supervised and unsupervised learning  
<cit.>.    The 
major approach to address negative transfer  for multi-task learning has been 
through task grouping where 
knowledge transfer
is performed only within each group 
<cit.>. Bakker et  al. for instance, presented a 
 
Bayesian approach in which some of the model parameters were shared   and 
others 
 loosely connected through a joint prior distribution   learnt from the data 
<cit.>.  Zhang and Yeung presented a convex formulation for 
multi-task metric learning by modeling the task relationships in the form of a 
task covariance matrix <cit.>.  Moreover, Zhong et  al. 
presented 
flexible 
multi-task learning framework to
identify latent grouping structures in order to restrict negative knowledge 
transfer <cit.>. Multi-task learning  has recently contributed to a 
number of successful 
real-world applications that gained better performance by exploiting shared 
knowledge for multi-task formulation.  Some of these applications  include 
 1)  multi-task 
approach for “ retweet” prediction behaviour of individual  users  
<cit.>, 2)
recognition of facial action units   <cit.>, 3) automated Human 
Epithelial Type 2  (HEp-2) cell 
classification   <cit.>, 4)   kin-relationship verification using 
visual 
features  <cit.> and 5)  object tracking     
<cit.>. 




 §.§ Cooperative Neuro-evolution
 

 
 Neuro-evolution  employs evolutionary algorithms for training 
neural networks <cit.> which can be classified 
into   direct 
<cit.>  and indirect encoding strategies 
<cit.>. 
In direct  encoding,   every
connection and neuron is specified directly and explicitly in the genotype 
<cit.>.  In  indirect  
encoding, the genotype specifies rules or some other structure for generating 
the network
<cit.>. Performance of direct and indirect encodings varies for 
specific problems.  Indirect encodings seem very intuitive and have biological 
motivations, however, in 
several cases they have shown not to outperform direct encoding strategies 
<cit.>. 

Cooperative coevolution for training neural networks is known as cooperative 
neuroevolution <cit.>. Although 
cooperative coevolution 
faced challenges in problem decomposition,  it showed promising features 
that included   modularity and diversity 
<cit.>. Further challenges  have been in 
area of credit assignment  for subcomponents <cit.>, 
  problem decomposition, and 
adaptation due to problem of  separability that refer to grouping 
interacting or highly correlated variables
<cit.>. In  cooperative neuro-evolution, problem decomposition  
has a major effect in the training and generalisation performance. Although 
several decomposition strategies have been implemented that vary for different 
network architectures, the two established  decomposition methods   are 
those on the synapse 
  <cit.>  and neuron level <cit.>. In synapse level, the  network is 
decomposed to its lowest 
level where each weight connection (synapse) forms a 
subcomponent <cit.>. In neuron level, 
the neurons in the network act as the reference 
point for each subcomponent <cit.>. They 
have shown good 
performance in pattern classification problems <cit.>. Synapse level  decomposition has shown good 
performance in control 
and time series prediction  problems 
<cit.>, however, they gave  poor 
performance for pattern classification problems  <cit.>. Chandra 
et  al. 
applied neural and synapse level   decomposition  for chaotic time series 
problems using recurrent neural networks 
<cit.>. Hence, it was established that synapse level encoding 
was 
more effective  for time series and control problems 
<cit.>.    Chandra later presented 
competition and collaboration with neuron and synapse decomposition strategies 
 during evolution which improved the performance further
<cit.>.

 


 
In Algorithm <ref>, the   network is decomposed according to 
the selected  decomposition method. Neuron level decomposition is shown 
in Figure <ref>. Once the decomposition is done, the subcomponents that 
are implemented as sub-populations are initialized and evolved in a round-robin 
fashion, typically for a fixed depth of search given by generations. 
The 
evaluation of the fitness of each individual for a particular sub-population is 
done cooperatively by concatenating the current individual with the fittest 
individuals from the rest of the sub-populations 
<cit.>.   The concatenated individual 
is then  encoded  into the neural 
network where its fitness is evaluated and returned. Although   it is a 
representative fitness, the 
fitness of the entire network is   assigned to the particular 
individual of the sub-population. This is further illustrated  in Figure 
<ref>.








  






 §.§ Dynamic programming and reinforcement learning

   Dynamic programming (also known as dynamic optimisation) is a 
optimisation strategy that considers breaking a large problem into sub-problems 
and using their solution as a building block to solve the 
bigger problem <cit.>.  By simply using  previously 
computed solution taken  from a sub-problem,   the paradigm  
improves the  computation time and  also  becomes efficient in memory or 
storage. 
 Although dynamic programming has typically been an approach for optimisation 
and sequential problems  <cit.>, it has been well used in the 
areas of machine learning (such as spoken word recognition 
<cit.>),  and  computer vision (such as 
variational problems <cit.>).   

   Reinforcement learning on the other hand,  considers agents 
that  take actions in an environment   to maximise the notion of cumulative 
reward <cit.>. Reinforcement learning has a wide range 
of multi-disciplinary applications  such as game theory, control theory, and
operations research <cit.>. In the operations 
research and control literature, reinforcement learning is called approximate 
dynamic programming <cit.>, or neuro-dynamic programming 
<cit.>. They combine ideas from the fields of neural
networks, cognitive science and approximation theory. In machine learning, the 
environment is typically formulated as a Markov decision process (MDP) where 
the outcomes are partly random and  partly under the control of a decision 
maker. Opposed to classical dynamic programming,  reinforcement learning 
  does not assume knowledge of an exact mathematical model of the MDP. 
  Recent application  via deep learning 
considers learning of policies directly from high-dimensional sensory 
inputs  in the challenging domain of classic Atari 2600 games 
<cit.>. 

   Evolutionary algorithms have been proposed as  a method for  
reinforcement learning  <cit.>. Reinforcement learning 
via evolutionary algorithms  have been implemented as neuroevolution 
<cit.> for   neural networks  with application for 
playing the game of 'Go'. Reinforcement learning 
more recently has been implemented with co-evolutionary algorithms  in a 
classical  control problem  that considers balancing  double inverted poles
<cit.>. This further motivates the methodology presented 
in this paper that provides a synergy between, dynamic programming, 
reinforcement learning and neuroevolution for co-evolutionary multi-task 
learning.
 


 

 §.§ Machine learning and optimisation

 
 
Essentially, machine learning  algorithms have three components that include  
representation, evaluation and optimisation. Representation is done in the 
initial stage when the problem is defined and form formulated. Representation 
considers the type of problem (classification, regression or prediction) and 
the 
evaluation  metrics such as squared error loss and classification performance. 
Representation also considers initialisation of the parameters, such as the 
weights of the neural networks  and hyper parameters such as the learning rate. 
In the case of cooperative neuroevolution, the representation component would 
consider the encoding of the network weights into the subpopulations and 
initialising them for evolution.  Evaluation and optimisation are components 
that 
iterate over time until a certain condition is met.  Machine learning can also 
be seen as a data driven optimisation process. The learning procedure can be 
seen as 
solving a core optimisation problem that 
optimises the variables or parameters of the model with respect to the given 
loss 
function. Evolutionary algorithms are typically considered as optimisation 
methods and their synergy with neural networks into neuroevolution can be 
viewed 
as a learning procedure. In this paper,    learning in neural networks is 
implemented using   co-evolutionary algorithms that  features elements from 
multi-task learning and dynamic programming. Moreover, learning is also 
referred 
to evolution in the context of neuroevolution. Bennett and  Parrado-Hernández   
in an introductory note to a special issue of a journal mentioned  that 
optimisation problems lie at the heart of most machine learning approaches 
<cit.>. They highlighted the need for dealing with 
uncertainty, convex models, hyper-parameters,  and hybrid approaches of 
optimisation methods for learning. Furthermore, Guillory et al. showed  
that  
online active learning algorithms can be viewed as stochastic gradient descent 
on non-convex objective functions <cit.>. 
  
 

 §.§ Problems in time series prediction 

  

  Although a  number of methods have been used for one-step ahead  prediction,  
 neural networks have given  promising results   with different architectures 
<cit.> and algorithms that include 
gradient-based learning <cit.>, evolutionary 
algorithms <cit.>,  and hybrid 
learning methods 
<cit.>. These methods can also be used 
for multi-step ahead and multivariate time series prediction. Multi-step-ahead  
(MSA) prediction refers to  the forecasting or  prediction of 
 a 
sequence of 
future values from  observed trend in a time series 
<cit.>.  It is challenging to develop models that 
produce low  
prediction error as the prediction horizon increases  
<cit.>. MSA
prediction has been approached mostly with the recursive and  
direct  strategies. In the recursive strategy, the prediction from 
a one-step-ahead prediction model is used as input for future
prediction horizon
<cit.>. Although 
relatively new, a third strategy is a combination of these approaches   
<cit.>. 


Multi-variate time series prediction typically  involves the prediction of 
single or multiple values from multi-variate input that  are 
typically  interconnected through some event 
<cit.>. Examples   of single value 
prediction  are the  prediction of flour  prices of time series obtained from   
different cities <cit.> and  traffic   time 
series <cit.>. The goal in this case  is  to enhance 
the prediction performance from the  additional features in the input, 
although the problem can be solved in a univariate approach <cit.>. In 
the case of prediction of multiple values, the model needs to predict future 
values of the different features, for example, prediction of 
latitude and longitude that defines the movement of cyclones 
<cit.>. A recent study has shown that     
multivariate 
prediction would perform  better than univariate for MSA  as the prediction 
horizon becomes larger, multi-variate information 
becomes more important 
<cit.>. Another area  of problems in  time series 
prediction 
 consist of  applications that have missing data.  Wu et  al. approached 
the 
missing data problem in time series with non-linear filters and neural networks
<cit.>.  In their method, a sequence of independent 
Bernoulli random variables  were used  to model random interruptions which was 
later  used to construct the state-space vector in pre-processing stage.  



 Furthermore, novel approaches  that feature  a synergy of 
different methodologies  have recently been presented to address time series 
prediction. Extreme  value 
analysis considers the  extreme deviations 
from the median of probability distributions, which has been beneficial for 
time series prediction in the past <cit.>. D'Urso et  
al. 
explored the grouping 
  of time series with similar seasonal patterns using extreme value 
analysis with fuzzy clustering   with an application   daily 
sea-level time series in Australia <cit.>.
Chouikhi et  al. presented echo state networks for time series prediction 
where
particle swarm optimised was used   to optimise the 
untrained weights    that gave  enhancement  to  learning  <cit.>. 
Such approaches give motivations for developing a synergy of different 
methods in order to utilise their strengths and eliminate their weaknesses.   



§ CO-EVOLUTIONARY MULTI-TASK LEARNING
   


 

 §.§ Preliminaries: time series reconstruction




State-space reconstruction   considers the use of Taken's theorem 
which expresses that the
state-space vector  reproduces  important
characteristics of the original time series  <cit.>. Given an 
observed time series x(t), an embedded  state
space Y(t) = [(x(t),
x(t-T),..., x(t - (D-1)T)] can be generated, where, T is the time delay, D 
is
the timespan (also known as embedding dimension), t=   (D-1)T,  DT, ..., N-1, 
and N is 
the 
length 
of the
original time series.    The optimal values for D and T
must be chosen in order to efficiently apply Taken's theorem 
<cit.>. Taken's proved that if the original attractor is of
dimension d, then D = 2d+1 will be sufficient to reconstruct the attractor
<cit.>. In the case of using feedforward neural  networks, D is 
the number of input neurons. 






 §.§ Dynamic time series prediction



 

Natural disasters such as torrential
rainfall, cyclones, tornadoes,  wave surges and 
droughts <cit.>  require 
dynamic and robust  prediction models that  can make a decision as soon as  the 
event take place. Therefore,  if the model is trained over specific  
months for rainy seasons, the system should be able to make a 
robust prediction from the beginning of the rainy season. We define the event length as the duration of an event which can be number of hours of a 
cyclone or number of days of drought or torrential  rain. 


As noted earlier, in  a typical time series prediction problem, the 
original time series is reconstructed using Taken's theorem 
<cit.>. In the 
case of cyclones, it is important to  measure  the performance of the  model 
when dynamic prediction is needed regarding track, wind or other 
characteristics of the cyclone <cit.>. Dynamic prediction can  
provide early warnings to the community at risk.  For instance, data about 
tropical cyclone in the South Pacific is recorded at six hour intervals 
<cit.>. If the   timespan D = 6, the first prediction 
by the model at hand   would come after 36 hours which could have devastating 
effects.

 The  problem arises  when  the gap between each data point in the times series 
is a day or number of hours. The problem with the existing models such as 
neural 
networks used for cyclones  is the minimal
timespan D needed to make a prediction. It has been  reported that 
 recurrent  neural 
networks trained with a given timespan (e.g. D=5), cannot make 
robust
prediction for other timespan ( e.g. D=7 or D=3 )
<cit.>. 
 Therefore, we introduce and define  the problem  of 
dynamic time 
series 
prediction   that refers to the ability of a model to give a 
prediction given a set of timespan values rather than a single one.  This 
enables the model to make decision with minimum value of the timespan in cases 
when rest of features of data-points are not available.   A conventional 
one-step ahead time series 
prediction can be given by 






    x   =   x[t], x[t- 1], ..., x[t - D] 
    
     xˆ[t+1]    = f(x)


where f(.) is a model such as a feedforward neural network and D is a fixed 
value for the  timespan and  x refers to the input features. In the 
case 
of 
dynamic time series, rather than a single value, we consider a set of values   
for the timespan 


    Ω_m =[D_1, D_2,...D_M]


where M is the number of subtasks, given  
M ≤ D. Hence,  the input features for each subtask in  
dynamic time series prediction can be 
given by Ψ_m, where m= 1, 2, ..., M. 
 



    Ψ_m    =   x[t], x[t- 1], ..., x[t - 
    Ω_m]
 



 §.§ Method


 
  
In the proposed method, a co-evolutionary algorithm based on a dynamic 
programming 
strategy is proposed for multi-task learning. It features  problem 
decomposition in a  similar way as cooperative coevolution, however, the major 
difference lies in the way the solutions of the subcomponents are combined 
 to build to the final solution. Hence, the proposed co-evolutionary multi-task 
learning algorithm is inspired from the strategies used in dynamic programming 
where a subset of the solution is used as the main building block for the 
optimisation problem. In this case, the problem is  learning 
the weights of 
a cascaded neural network architecture where  the base problem is the   network 
module that is defined by lowest number of input features  and  hidden neurons. 


The weights in the base 
network are part of larger cascaded network modules that consist of additional 
hidden neurons and input features. This can be viewed as modules of knowledge 
that are combined for larger subtasks that use knowledge from smaller subtasks 
as 
building blocks. The cascaded network architecture can also be viewed as  an 
ensemble of neural networks that feature distinct topologies in terms of number 
input and hidden neurons as shown in Figure <ref>. Suppose that we 
refer to a module  in the cascaded ensemble, there are  M modules 
with  input i, hidden h  layers  as shown.  
 



    I     = [i_1, i_2, ...,i_M] 
     
     H    = [h_1, h_2, ...,h_M] 
     
      O    = [1, 1, ..., 1]


where I, H, and O contain the set of input, hidden and 
output layers.  Note that the 
approach considers fixed number of output neurons. Since 
we consider one-step-ahead time series problem, one neuron in output layer is 
used for all the respective modules. The input for each of the modules is given 
by the dynamic nature of the problem that considers different lengths of 
timespan that constructs an input vector  Ψ_m for the given module as 
follows. 
 


    Ψ_m = x[t], x[t- 1], ..., x[t - I_m]

 
 Note that the input-hidden layer ω_m weights  and the 
hidden-output layer υ_m weights are combined for the respective module 
m. The base knowledge module is given as Φ_1  = [ω_1, υ_1]. 
The 
subtask  θ_m  is defined as the  problem of training the 
respective  
knowledge 
modules Φ_m  with given input Ψ_m.   Note 
that 
Figure  <ref>  explicitly shows the knowledge modules
of the network   for ω_2 and ω_1, 
respectively. The  knowledge module for each  subtask is constructed in a 
cascaded network architecture 
as follows.  

    Φ_1     = [ω_1, υ_1];   θ_1=(Φ_1)
    Φ_2   =[ ω_2,υ_2 ];  θ_2 = [θ_1,Φ_2]
       ⋮
    Φ_M    =  [ ω_M,υ_M ];  θ_M=[θ_M-1,Φ_M]

The   vector of knowledge modules considered for training or optimisation  is 
therefore
Φ=(Φ_1,…,Φ_M). 

  



    y_1    = f(θ_1, Ψ_1 ) 
    
      y_2    = f(θ_2, Ψ_2 ) 
       ⋮
    
      y_M    = f(θ_M, Ψ_M )



Given T samples of data, the loss L for sample t can be calculated by 
root mean squared error.

    L_t  = √(1/M∑_m=1^M 
    (ŷ  -y_m)^2  )


where ŷ is the observed time series and y_m is the 
prediction given by   subtask m. The loss E for the entire 
dataset (all subtasks) is given by 


    E  = 1/T∑_t=1^T L_t



 The training of the cascaded network architecture involves  
decomposition  as subtasks through    co-evolutionary 
multi-task learning  (CMTL) algorithm.  The knowledge modules in subtasks 
denoted  by Φ_m are implemented as  subcomponents    S_1, 
S_2, ..S_M, where M is number of 
subtasks.   The subcomponents are implemented as 
sub-populations consist of matrix of variables 
that feature  the weights and biases S_m = Π_i,j, where   i refers 
to  weights and biases 
and j refers to the  individuals. The individuals of the sub-populations 
are 
referred as genotype while  the 
corresponding network module are referred as  the phenotype. Unlike conventional 
transfer learning methods,  the transfer 
of knowledge here is done implicitly through the sub-populations in CMTL. The 
additional subtasks are implemented through the cascades that utilise knowledge 
from the base subtask.  The fitness of the cascade is evaluated by utilising 
the knowledge from the base subtask. This is done through CMTL where the best 
solution from the sub-population of the base subtask is concatenated with the 
current individual from the sub-population whose fitness needs to be evaluated. 
This is how transfer of knowledge is implicitly done through co-evolutionary 
multi-task learning. 
 
 
 
 
 
 

 
 
 
Algorithm <ref> gives details for CMTL which begins by 
initialising  the   the sub-populations   
defined by the subtasks   which feature  the knowledge modules
Φ_m and 
respective subtask input features Ψ_m. The sub-populations   are 
initialised 
with real 
values  [-α,α] drawn from uniform distribution where 
α defines the range.   Once this has been done, the algorithm moves into 
the 
evolution phase where each subtask is evolved for a fixed number for 
generations 
defined by depth of search, β. The major  concern here is the 
way the phenotype is mapped into genotype where a group of weight matrices 
given by Φ_m  =[ω_m, υ_m] that makes up subtask θ_m are 
converted into vector X_m. Stage 1 in Algorithm <ref> implements the  
use of  knowledge from 
previous subtasks through multi-task learning.  In the case if the  subtask is 
a base problem (m 
==1), then the subtask solution X_m is utilised in a conventional matter 
where knowledge from other subtasks or modules are not required to reach a 
decision. Howsoever, given 
that the subtask is not a base problem,   the 
current  subtask individual X_m is appended with best individuals
from the previous subtasks, therefore,   X_m = [B_1,...,  B_m-1, V_m], 
where B is the best individual from previous subtask and V is the current 
individual that needs to be evaluated. This will encode  
X_m into knowledge
modules Φ=(Φ_1,…,Φ_M) for the respective subtasks. The 
algorithm then calculates subtask network output or prediction 
y_m = f(θ_m, Ψ_m)  and evaluate  the  individual  
though the loss function  E  = 1/T∑_t=1^T L_t, where L_t 
is given in Equation <ref>. The subtask 
solution is passed to Algorithm <ref> along with the network topology   
in order to decode the subtask 
solution 
into the 
respective weights of the network. This could be seen as the process of  
genotype to phenotype mapping.  This procedure is executed for every 
individual  in the sub-population.  This 
procedure 
is 
repeated for every sub-population   for different phases until the termination 
condition 
is 
satisfied. The termination condition can be either the maximum number of 
function evaluations or a minimum fitness value from the training or validation 
dataset. Figure <ref> shows an exploded view of the neural 
network topologies associated with the respective  subtasks, however, they are 
part of the cascaded  network architecture  later shown in Figure 
<ref>. The  
way the subtask solution is decomposed and mapped into the network is given in  
Figure <ref> and discussed
detail in the next section.  
 
 
 The major difference in the implementation of CMTL when compared to  
conventional  cooperative neuroevolution  (Algorithm <ref>) is by 
the way the problem is decomposed and the 
way the fitness for each individual is calculated. CMTL is motivated by dynamic 
programming approach where the best solution from previous sub-populations are 
used for cooperative fitness evaluation  for individuals in the current  
sub-population.  Howsoever, the current sub-population  does not use the 
best solution from future subpopulations. This way, the concept of utilising
knowledge from previous subtasks as building blocks is implemented . On the 
other hand, cooperative neuroevolution follows a divide and conquer approach 
where at any given  subpopulation, in order to evaluate the individuals,  the 
best individuals from the rest of the sub-populations are taken into account. 
     






Finally, when the termination criteria has been met, the algorithm moves into 
the testing phase where the best solutions from all the different subtasks are 
saved and encoded into their respective network topologies. Once this is done, 
the respective subtask test data is loaded and the network makes prediction 
that is evaluated with loss  
E given in Equation <ref>. Other measure of error can also be 
implemented. Hence, we have highlighted the association of 
every individual in the respective sub-populations  with  different subtasks in 
the multi-task learning  environment. There is transfer of knowledge in terms 
of weights from smaller to bigger networks as defined by the subtask with its 
data 
which is covered in detail in next section.  A Matlab   implementation of 
this algorithm with respective datasets used for the experiments  is 
given online 
[https://github.com/rohitash-chandra/CMTL_dynamictimeseries
https://github.com/rohitash-chandra/CMTL_dynamictimeseries].


 



 §.§ Transfer of knowledge


One challenging aspect of the Algorithm <ref> is the transfer of 
knowledge represented by the  weights of the respective neural networks that is 
learnt  by the different subtasks in CMTL.  The cascading network architecture 
 increase in terms of the input and hidden neurons  with the subtasks. 
Algorithm <ref>  implements transfer of knowledge given the changes of 
the 
architecture by  the different subtasks. The goal  is to transfer  weights 
that are mapped from different sub-populations defined by the subtasks.  The 
algorithm  is 
given 
input parameters which are 

 

  * The   reference to subtask m;  
 

  * The current subtask solution; if m is base task,  X_m = [V_m] 
else      X_m = [B_1,..., B_m-1, V_m ], otherwise.  

  * The topology of the respective cascaded neural module  for the 
different subtasks in terms of number of  input, hidden, and output neurons. 

 
We   describe the algorithm with reference to Figure <ref> 
which shows a case, where the subtask m = 3 goes through the 
transfer where  m = 1 and m = 2 are used as building blocks of 
knowledge 
given  in the weights. Therefore, we use examples for the network topology 
as highlighted below. 

 
 
  * I_m is vector of number of input neurons for   the  
respective subtasks, eg. I = [2, 3, 4 ] ;
  
  * H_m is vector of number of input neurons for  the respective 
subtasks, 
eg. H_ = [2, 3, 4];

  * O_m   is vector of  output neurons for   the respective subtasks, 
eg. 
O = [1, 1, 1].  



 The algorithm begins by assigning base case,  b = 1 which is 
 applied irrespective of the number of subtasks. In Step 1, the transfer 
of  Input-Hidden layer weights  is   shown by weights (1-4)  in Figure 
<ref>.  Step 2 
executes the transfer for   Hidden-Output layer weights  as  
shown by weights (5-6)  in  Figure 
<ref>. Note that Step 1 and 2 are applied for all the cases given by 
the number of subtasks. Once this is done, the algorithm terminates if  
m = 
1 or proceeds if m >= 2. Moving on, in Step 3, the 
case is more complex as we consider m >= 2.   Step 1 
and 
2 are executed before moving to Step 3 where X contains the 
appended  solution sets from previous subtasks. In  Step 3,  t in principle  
points to the 
beginning of the solution given by sub-population for m = 2. Here, the  
transfer for  Input-Hidden layer weights  (7-9)  is executed 
for m = 2. Note that in this case, we begin with the weights with reference 
to the number of hidden neurons from previous subtask j= H_(m -1) 
+ 1, 
and move to the number of hidden neurons of  the current subtask j= 
H_(m)  in order to transfer the weights   to all the input 
neurons. This refers to  weights  (7-9) in Figure <ref>. 
Before 
reaching    transfer for m = 3, m = 1 and m = 2 
transfer 
would have already taken place and hence the weights (13-16)  would be 
transferred as shown in the same figure. Moving on to Step 4,  we first 
consider the transfer for  Input-Hidden 
layer 
weights  for m = 2  through the transfer of weights  from beginning of 
previous subtask input,  i=I_(m-1)+1  to   current subtask 
input connected with all hidden neurons. This is   given by weights 
(10-11) in 
Figure <ref>. For the case of m = 3, this would refer to 
weights (17-19) in the same figure. 


  Finally, in Step 5, the algorithm executes the transfer for   Hidden-Output 
 layer weights based on the  hidden neurons from 
previous subtask. 
In 
case of m = 2,  this results in transferring weight (12) and 
for 
m = 
3, the transfer is  weight (20) in Figure <ref>, 
respectively. Note that the algorithm can transfer any number of input and 
hidden neurons as 
the  number of  subtasks  increase.  





 
 









 
 The time complexity of CMTL   considers the time 
taken for transfer of solutions for different number of subtasks. We
note that the best case is when the subtask is the base subtask (m = 1). 
Therefore, the worst case time complexity can be 
given by 
 



    T(m<=1)    = O(1) 
    
     T(m)    = T(m-1) + T(m-2), ...,  + O(1)
    
     T(m)    = O(2^m)


where m refers to the subtasks. 
 




 


§ SIMULATION AND ANALYSIS


This section presents an experimental study that compares the performance of 
CMTL with conventional neuroevolution methods.    The results are compared 
with  neuroevolution via evolutionary algorithm (EA)  and 
cooperative neuroevolution (CNE)  for benchmark time series prediction 
problems. Furthermore,  tropical 
cyclones from South Pacific and South Indian Ocean are considered  to address 
the minimal timespan issue  <cit.> using dynamic 
time series prediction. 


 

 §.§ Benchmark Chaotic Time Series Problems  
 

In the benchmark chaotic time series problems, the Mackey-Glass, Lorenz, Henon 
 and Rossler are the four synthetic time series problems. The experiments use 
the chaotic time series with length of 
1000 generated by the respective chaotic attractor.  The first 500 samples are 
used for training and the remaining  for testing.

 In 
all cases, the phase space of the original time series 
is reconstructed  with the timespan for 3 datasets for the 
respective subtasks with the set of timespan Ω = [3, 5, 7] and time 
lag T = 2. All the synthetic and  
real-world  time series were 
scaled in the range [0,1]. Further details of each of the time series problem 
is given as follows. 

The  Mackey-Glass time series    has been used in literature as a 
benchmark problem due to its chaotic nature <cit.>. The 
Lorenz time series  was introduced by Edward Lorenz  who has
extensively contributed to the establishment of Chaos theory 
<cit.>. The Henon time series  is generated with a Henon 
map which is a discrete-time dynamical system  that exhibit chaotic behaviour 
<cit.> and the Rossler time series  is generated using the 
attractor for the Rossler system, a system of three non-linear ordinary 
differential equations as given in <cit.>. The real-world problem  are 
the Sunspot, ACI 
finance  and Laser time series. The Sunspot time series is a good 
indication of the solar activities for solar cycles which impacts Earth's 
climate, weather patterns, satellite and space missions <cit.>.   
The Sunspot time series from November 1834 to June 2001 is selected which 
consists of 2000 points.     The ACI financial time series  is obtained 
from the ACI
Worldwide Inc.  which is one of the companies
listed on the NASDAQ stock exchange.  The data set contains
closing stock prices from December 2006 to February 2010,
which is equivalent to approximately 800 data points.  The closing
stock prices were normalised between 0 and 1. The data set
features  the recession that hit the U.S. market in 2008   
<cit.>. The Laser time series  is   measured in a 
physics laboratory experiment that were used in the Santa Fe 
Competition <cit.>. All the real world time series used the first 
50 percent samples for training and remaining for testing. 
 
  
 

 §.§ Cyclone time series 


 The Southern Hemisphere tropical cyclone best-track data from 
Joint Typhoon 
Warning Centre  recorded every 6-hours is  used as the main source of data 
<cit.>.  We consider only the austral summer tropical 
cyclone season 
(November to April) from 1985 to 2013 data  in the current study as data prior 
to the satellite era is  not reliable due to inconsistencies and missing 
values. The original data of tropical cyclone wind intensity in the South 
Pacific 
  was divided into training  and testing set as follows:  
 
 
 
  * Training Set: Cyclones from 1985 - 2005 (219 Cyclones with 6837 data 
points) 
 
  * Testing  Set: Cyclones from 2006 - 2013  (71 Cyclones with 2600 data 
points)
  
  
In the case for South Indian Ocean, the details are  as follows: 
 
 
 
  * Training Set: Cyclones from 1985 - 2001 (  285 Cyclones with 9365 data 
points) 
 
  * Testing  Set: Cyclones from 2002 - 2013  (  190 Cyclones with 8295 data 
points )
  
  
  Although the cyclones are separate events,   we choose to  combine 
all the 
cyclone data  in a consecutive order as given by their 
date of occurrence. The time series 
is reconstructed  with the set of  timespan,   Ω = [3, 5, 7], and 
time 
lag   T = 2.  

  
 

 §.§ Experimental Design
 
 

  We note that multi-task learning approach used 
for dynamic time series can be formulated as a series of independent single 
task learning approaches. Hence, for comparison of CMTL,  we provide 
experimentation and results with conventional  neuroevolution methods that 
can be considered as single-task learning   (CNE 
and EA). In the case of CNE, neuron level problem decomposition 
is applied for training   
feedforward networks <cit.>.   We employ 
covariance matrix adaptation evolution 
strategies (CMAES) 
<cit.> as the evolutionary algorithm in sub-populations of CMTL, 
CNE and the population of EA.  The  training and generalisation 
 performances are reported for each case given by the different subtasks in the 
respective time series problems.




The respective neural  networks used both  sigmoid units in the hidden and 
output layer 
for all the different problems.   The loss function given in Equation 
<ref>  is used as the main performance measure.  Each neural 
network architecture was tested with different numbers of hidden neurons.  
 

We employ fixed depth = 5  generation in the sub-populations of CMTL as the 
depth
of search as it has given  optimal performance in trial runs. CNE also employs 
the same value.   Note that all the sub-populations   evolve for the same depth 
of search. The population size of CMAES in the respective methods is given 
by P = 4+ floor*(3*log(W)), where W is the total number of weights and 
biases   for the 
entire neural network  that includes all the subtasks (CMTL). In the case of EA 
and CNE,  W is total number of weights and biases for the given network 
architecture.

The 
termination condition is fixed at 30 000 function evaluations for each subtask, 
hence, CMTL employs 120 000 function evaluations while  conventional methods
use 30 000 for each of  the respective subtasks for all the problems. Note that 
since there is a fixed training time, there was no validation set used to stop 
training. The choice of the parameters such as the appropriate 
population size, termination condition has been determined  in trial 
experiments. The experiments are well aligned with  experimental setting  
 from
previous 
work  <cit.>.

 



 §.§ Results for Benchmark Problems

 
   The results for the 7 benchmark chaotic time series problems are given in 
Figure 
<ref> to <ref> which highlight the training  and 
generalisation performance.  We limit our discussion to  
the 
generalisation performance, although the training performance is also 
shown. Figure <ref> shows that CMTL generalisation  performance is 
better 
than  EA and CNE, while  CMTL and EA outperform   CNE in all the subtasks 
denoted by the timespan. The same trend 
is shown in general  for Lorenz and Henon time series as shown in Figure 
<ref> and Figure <ref>, respectively. There is one 
exception (D=5), for the Henon problem  where CME gives better performance 
than 
EA, however, worse than CMTL. Figure <ref> shows the results for 
the Rossler time series which follows a similar trend when compared to the 
previous problems. Hence, in general,  CMTL 
generalisation performance is the best when compared to the conventional
methods (CNE and EA) for the 4 synthetic
time series problems which have little or no   noise present. 

In the case of real-world problems, Figure <ref> for 
the Sunspot problem shows that CMTL provides the best generalisation 
performance when compared to EA and CNE for all the cases. The same 
is given for first two timespan cases for ACI-finance problem as shown in 
Figure <ref>, except for one case  (D=7), where EA  and CMTL gives 
the same performance. In the case of the Laser time series in Figure 
<ref>, which is known as one of the most chaotic time series 
problems, CMTL outperforms CNE and EA, except for one case, D=7. Therefore, 
at this 
stage, 
we can conclude that CMTL gives the best performance for most of the cases in 
the real-world time series problems. Table <ref> shows the mean of 
RMSE and confidence interval  across 
the 3 timespan. We find that the CMTL performs better than EA 
and CNE for almost  all the problems. The  Laser problem is  the only 
exception where the EA is slightly better than CMTL.




 























 









 §.§ Results for Tropical Cyclones

 We present the results for the performance of the given methods 
 on the 
two selected cyclone problems, which features  South Pacific and South Indian 
ocean as shown in Figure  <ref> and Figure <ref>, 
respectively. Figures <ref> and <ref> show the 
prediction performance of a typical experimental run.
 In the case of the South Pacific ocean, the results show that 
CMTL provides the 
best generalisation  performance when compared to CNE and EA. This is 
also observed for the South Indian ocean. 

 







  

 


 

 

 §.§ Discussion

 
   The goal of the experiments was to evaluate if  CMTL can maintain quality in 
prediction performance when compared to conventional methods, while at the same 
time address dynamic time series problems. The results have shown that CMTL  not 
only addresses dynamic time series but is a  way to improve the performance if 
each of the subtasks in multi-task learning was decomposed and approached as 
single-task learning. The incremental learning in CMTL not only improves the 
prediction performance but also ensures modularity in the organisation of 
knowledge. Modularity is an important attribute for addressing dynamic 
prediction problems since groups of knowledge can be combined to make a decision 
when  the nature or complexity of the problem increases. Modularity is important 
for design of neural network in hardware  <cit.> as 
disruptions in certain synapse(s) can result in problems with the whole network 
which can be eliminated by persevering knowledge as modules <cit.>.  

It is noteworthy that CMTL gives consistent performance even for the cases when 
the problem is harder as in the case of smaller or foundational subtasks that 
have minimal timespan.  Learning smaller networks could be harder since they 
have limited information about the past behaviour of the time series. The way 
the algorithm handles this issue  is with the refinement of the solutions (in a 
round-robin manner through coevolution) after it has been transferred to a 
larger network. When more information is presented as the subtask increases, 
CMTL tends to refine the knowledge in the smaller subtasks. In this way, the 
results show that the performance is consistent given the small subtasks and 
when they increase depending on the timespan. 
 


 CMTL  can be seen 
as a flexible method for datasets that have different features of which some 
have properties so that they can be grouped together as subtasks.  Through 
multi-task learning, the overlapping features can be used as building blocks to 
learn 
the nature of the problem through the model at hand. Although feedforward 
neural networks have been used in CMTL, other   other neural network 
architectures and learning models can be used depending on the nature of 
the subtasks. In case of 
computer vision applications such as face recognition, the different subtasks 
can 
be different number of features; i.e.  the algorithm  can execute  face 
recognition based on minimal features from a set of features.



 CMTL could  also be viewed as   training cascaded 
networks  using a dynamic programming approach where each cascade defines a 
subtask. Although the 
depth 
of the cascading does not have a limit,  adding cascades 
could result in 
an exponential increase of training time.  The depth of the cascaded 
architecture would 
be dependent on the application problem. It depends on the time series under 
consideration and the level of inter-dependencies of the current and  the 
previous time steps. In principle, one should stop adding cascades when the 
prediction performance begins to deteriorate to a given threshold.  Therefore, 
for a given problem, there needs to be systematic approach that selects the 
number 
of subtasks for the cascaded architecture.


 We have experimentally tested robustness and scalability 
using the synthetic and real-world datasets that includes benchmark problems 
and an
application  that considers the prediction of wind-intensity in tropical 
cyclones. We provided comprehensive experimentation  for the algorithm 
convergence given a range of initial  conditions. These include different set 
of   
initialisation of the subpopulations in CMTL with multiple experimental runs  
along 
with further reporting of the mean and confidence interval. We evaluated the 
prediction capability given different instances of the timespan 
defined as subtasks in CMTL and compared the performance with standalone 
methods. The experimental design 
considered multiples experiment runs, different and distinct datasets, 
difference in
size of the datasets, and different sets of initialisation in the 
subpopulations. 
In this way, we have addressed robustness and convergence of the proposed 
method, experimentally.  
 
 
 In the comparison of CMTL  with CNE, we observed that CMTL creates a 
higher time complexity since it has an additional step of transfer of solutions 
from the different subtasks encoded in the subpopulations. The time taken would 
increase exponentially as the number of subtasks increases. This would add to 
cost of utilising solutions from other subtasks given a fixed convergence 
criteria defined by the number of function evaluations. In case if the 
convergence 
criteria is defined by a minimum error or loss, it is likely that the solutions 
from the previous subtasks will help in faster convergence. In terms of 
scalability, we note that neuroevolution methods have limitations due to slow 
convergence of evolutionary algorithms. With help of gradient-based local 
search 
methods, convergence of  CMTL can be improved  via  memetic  algorithms where 
local refinement occurs during the evolution <cit.>. There is scope in 
future work convergence proof used in standard evolutionary algorithms   
<cit.>    that could further be extended for multiple sub-populations 
in CMTL.   
  
 


§ CONCLUSIONS AND FUTURE WORK
  


We presented a novel algorithm that provides a synergy between coevolutionary 
algorithms  and 
multi-tasking for dynamic time series problems. CMTL can be used to train a 
model 
that can handle multiple 
timespan values that defines dynamic input features which provides dynamic  
prediction.   CMTL has been 
very beneficial  for  tropical 
cyclones where timely prediction needs to be made as soon as the event takes 
place.   The results 
show that CMTL  addresses  the 
problem of   dynamic time series and provides  a  robust way to improve the 
performance  when compared to related  methods. 



It is important to understand how CMTL achieved better results for when 
compared 
to related  methods  given   the same neural network topology and data for 
respective subtasks.   CMTL can be seen as an incremental evolutionary learning 
method that features 
 subtasks as building blocks of knowledge. The larger  subtasks take 
advantage of 
knowledge 
gained  from learning the smaller subtasks.  Hence, there is diversity in 
incremental knowledge  
development  from the base subtask which seems to be beneficial for 
future subtasks. However, the reason why the base subtask produces better 
results 
when 
compared to  conventional learning   can be explored with further analysis 
during   learning. The   larger subtasks  with 
overlapping features covers the 
base subtask in a cascaded manner. Therefore,  larger subtasks can be seen as 
those that have 
additional 
features that guide  larger  network  with more hidden neurons  during 
training. 
 
 Finally, CMTL is a novel approach that provides synergy of  a 
wide range of fundamental methods that include, dynamic programming, 
reinforcement learning, multi-task learning,  co-evolutionary algorithms and  
neuro-evolution. This makes the CMTL useful for some of the   applications where 
the mentioned fundamental methods have been successful. Since reinforcement 
learning has been utilised in deep learning, the  notion of 
reuse of knowledge as building bocks by CMTL could be applicable in areas of 
deep learning. In future work, apart from feed-forward networks, the idea of 
dynamic time series prediction that employs transfer and multi-task learning 
could be extended to other other areas that have simpler model representation, 
such 
as autoregressive models. Furthermore, CMTL can be used for other  
problems that can be broken into multiple subtasks,  such as  multiple step 
ahead and multivariate time series  prediction.   Although the paper explored 
timespan for univariate time series, the approach could be extended to  
pattern classification problems that involves large scale features. It can be 
extended for heterogeneous pattern classification problems where the dataset 
contains 
samples that have missing values or features.  CMTL can also be 
extended for transfer 
learning problems that can include both heterogeneous and homogeneous domain 
adaptation cases. In case of tropical cyclones, a multivariate approach can be 
taken where  the different subtasks can be seen as   features that include 
cyclone tracks, seas surface temperature, and humidity. 
 

§ REFERENCES

 IEEEtran
  




  
 

