1

Born to Learn: the Inspiration, Progress, and Future
of Evolved Plastic Artificial Neural Networks

arXiv:1703.10371v2 [] 6 Dec 2017

Andrea Soltoggio, Kenneth O. Stanley, Sebastian Risi

Abstract—Biological plastic neural networks are systems of
extraordinary computational capabilities shaped by evolution, development, and lifetime learning. The interplay of these elements
leads to the emergence of adaptive behavior and intelligence.
Inspired by such intricate natural phenomena, Evolved Plastic
Artificial Neural Networks (EPANNs) use simulated evolution
in-silico to breed plastic neural networks with a large variety
of dynamics, architectures, and plasticity rules: these artificial
systems are composed of inputs, outputs, and plastic components
that change in response to experiences in an environment. These
systems may autonomously discover novel adaptive algorithms,
and lead to hypotheses on the emergence of biological adaptation.
EPANNs have seen considerable progress over the last two
decades. Current scientific and technological advances in artificial
neural networks are now setting the conditions for radically
new approaches and results. In particular, the limitations of
hand-designed networks could be overcome by more flexible
and innovative solutions. This paper brings together a variety
of inspiring ideas that define the field of EPANNs. The main
methods and results are reviewed. Finally, new opportunities and
developments are presented.
Index Terms—Artificial Neural Networks, Lifelong Learning,
Plasticity, Evolutionary Computation.

I. I NTRODUCTION
Over the course of millions of years, evolution has led to the
emergence of innumerable biological systems, and intelligence
itself, crowned by the evolution of the human brain. Evolution,
development, and learning are the fundamental processes that
underpin biological intelligence. Thus, it is no surprise that
scientists have tried to engineer artificial systems to reproduce
such phenomena (Sanchez et al., 1996; Sipper et al., 1997;
Dawkins, 2003).
This endeavor is the domain of fields such as artificial
intelligence (AI) and artificial life (AL) (Langton, 1997) that
seek the emergence of intelligence and life from artificially
designed computation: the idea is to abstract the principles
from the medium, i.e., biological or in-silico, and utilize such
principles to devise algorithms and devices that reproduce
properties of their biological counterparts. A possible way to
design complex and intelligent systems is to simulate natural
evolution in-silico in the field of evolutionary computation
(Holland, 1975; Eiben and Smith, 2015). Sub-fields of evolutionary computation such as evolutionary robotics (Harvey
et al., 1997; Nolfi and Floreano, 2000) and neuroevolution
(Yao, 1999) specifically research algorithms that, exploiting
Department of Computer Science, Loughborough University, LE11 3TU,
Loughborough, UK, a.soltoggio@lboro.ac.uk
Department of Computer Science, University of Central Florida, Orlando,
FL, USA, kstanley@cs.ucf.edu
IT University of Copenhagen, Copenhagen, Denmark, sebr@itu.dk

artificial evolution of physical and neural models, seek more
autonomous design methods and principles towards artificial
intelligence. One assumption is that intelligence in biological
organisms relies considerably on powerful and general learning algorithms, designed by evolution, which are executed
during both development and throughout life. Thus, research
in neuroevolution, and more in general in AI, is progressively
moving towards the design, or the evolution, of lifelong online
learning neural systems (Coleman and Blair, 2012), rather than
static systems.
This paper reviews and organizes the field that studies
evolved plastic artificial neural networks, and introduces the
acronym EPANN. EPANNs are evolved because parts of
their design are determined by an evolutionary algorithm;
they are plastic because they are specifically designed to
undergo lifetime changes, e.g., in the connectivity among
neurons, at various time-scales while experiencing sensorymotor information streams. Lifetime learning is manifested
through continuous adaptation to often dynamic scenarios.
The final capabilities of such networks are determined by the
combination of evolved genetic instructions and continuous
learning that takes place as the network interacts with an
environment.
EPANNs are motivated by the promise of discovering
principles of neural adaptation, learning, and memory, and are
not limited to performance optimization on limited datasets.
This ambitious endeavor also entails a number of research
problems. One problem is the interaction of dynamics across
the evolutionary and learning dimensions, which make the
whole system difficult to design and analyze. An open question
is what are the abstractions that capture essential computational principles and enable intelligence. One further problem
is the high computational cost required to simulate even simple
models of lifetime learning and evolution. Finally, experiments
to evolve adaptation and intelligence are difficult to benchmark
because they aim to achieve loosely defined objectives such
as increasing behavioral complexity, intelligence, adaptability,
and evolvability (Miconi, 2008). For these reasons, EPANNs
explore a much larger search space and rarely compete with
machine learning algorithms specifically designed to improve
performance on well-defined and narrow problems.
The power of EPANNs, however, derives from two autonomous search processes: evolution and learning, which
arguably place them among the most advanced AI and machine
learning systems in terms of open-endedness, potential for
discovery, and human-free design. These systems rely the least
on pre-programmed instructions because they are designed to
autonomously evolve while interacting with a real or simu-

2

lated world. Plastic networks, in particular recurrent plastic
networks, are known for their computational power, which has
also been hypothesized to go beyond that of Turing Machines
(Cabessa and Siegelmann, 2014): evolution can be a valuable
tool to explore the power of those computational structures.
In recent years, progress in a number of relevant areas
has set the stage for renewed advancement of EPANNs:
neural networks, in particular deep networks, are becoming increasingly more successful and used; there has been
a remarkable increase in available computational power by
means of parallel GPU computing and dedicated hardware; a
better understanding of search, complexity, and evolutionary
computation allows for less naive approaches; and finally,
neuroscience and genetics provide us with an increasingly
large set of inspirational principles. This progress has changed
the theoretical and technological landscape in which EPANNs
first emerged, providing greater research opportunities than in
the past.
This paper aims to delineate a unique research field, firstly
defining the inspirational principles of EPANNs (Section II).
The body of research that advanced EPANNs is brought
together and described in Section III. Finally, this paper
describes research directions that are creating opportunities
and challenges (Section IV).
II. I NSPIRATION
EPANNs are strongly inspired by the biological evidence of
natural computation and intelligence (Floreano and Mattiussi,
2008; Downing, 2015). From a computational perspective, a
primary focus is to understand what are the fundamental ingredients and design principles discovered by nature that resulted
in high levels of adaptation and intelligence. Those principles
might then be abstracted and implemented in original ways insilico to achieve similar or better results to those observed in
nature. Crucially, artificial systems do not need to implement
the same constraints and limitations as biological systems
(Bullinaria, 2003). Thus, inspiration is not simple imitation.
We now know that genetic instructions determine the ultimate capabilities of a brain (Deary et al., 2009; Hopkins
et al., 2014). Different animal species manifest different levels
of skills and intelligence because of their genetic blueprint
(Schreiweis et al., 2014). The intricate structure of the brain
emerges from one single zygote cell through a developmental
process (Kolb and Gibb, 2011), which is also strongly affected
by input-output learning experiences throughout early life
(Hensch et al., 1998; Kolb and Gibb, 2011). Yet high levels
of plasticity are maintained throughout the entire lifespan
(Merzenich et al., 1984; Kiyota, 2017). These dimensions,
evolution, development and learning, also known as the phylogenetic (evolution), ontogenetic (development) and epigenetic
(learning) (POE) dimensions (Sipper et al., 1997), are essential
for the emergence of biological plastic brains.
The POE dimensions lead to a number of research questions. Can artificial intelligence systems be entirely engineered
by humans, or do they need to undergo through a less humancontrolled process such as evolution? Do intelligent systems
need to learn, or could they be born already knowing? Opinions and approaches are diverse. EPANNs assume that both

evolution and learning, if not strictly necessary, are conducive
to the emergence of artificial intelligence. While artificial
evolution is justified by the remarkable achievements of natural
evolution, the role of learning has gathered significance in
recent years. We are now more aware of the high level of brain
plasticity, and its impact on the manifestation of behaviors
and skills (LeDoux, 2003; Doidge, 2007; Grossberg, 2012).
Recent developments in machine learning (Michalski et al.,
2013; Alpaydin, 2014) and neural learning (Deng et al., 2013;
LeCun et al., 2015; Silver et al., 2016), highlight the importance of learning from large input-output data and extensive
training. Other areas of cognition such as the capabilities to
make predictions (Hawkins and Blakeslee, 2007), to establish associations (Rescorla, 2014) and to regulate behaviors
(Carver and Scheier, 2012) are also based on learning from
experience. Interestingly, skills such as reading, playing a
musical instrument, or driving a car, are mastered even if
none of those behaviors existed during evolutionary time, and
yet they are mostly unique to humans. Thus, human genetic
instructions have evolved not to learn specific tasks, but to
synthesize recipes to learn a large variety of general skills.
We can conclude that the evolutionary search of learning
mechanisms in EPANNs tackles both the long-running nature
vs. nurture debate (Moore, 2003), and a fundamental and
fast growing research area in AI. EPANNs’ inspiration, and
consequently this review, focus on evolution and learning,
and less on development because that may be seen as a form
of learning if affected by sensory-motor signals. We refer to
Stanley and Miikkulainen (2003a) for an overview of artificial
developmental theories.
Whilst the range of inspiring ideas is large and heterogeneous, the analysis in this review proposes that EPANNs build
upon the following broad principles:
• natural and artificial evolutionary processes,
• plasticity in biological neural networks,
• plasticity in artificial neural networks, and
• lifetime learning environments.
Figure 1 graphically illustrates the inspiration areas and the
particular research topics and fields that make the structure of
this review and define EPANNs.
A. Evolutionary processes
A central idea in evolutionary computation (Goldberg and
Holland, 1988) is that evolutionary processes, similar to those
that occurred in nature during the course of billions of years
(Darwin, 1859; Dobzhansky, 1970), can be simulated with
computer software. This idea led to the belief that intelligent
computer programs could emerge with little human intervention by means of evolution in-silico (Holland and Reitman,
1977; Koza, 1992; Fogel, 2006).
The emergence of evolved intelligent software, however, did
not occur as easily as initially hoped. The reasons for the slow
progress are not completely understood, but a number of problems have been identified, likely related to the simplicity of the
early implementations of evolutionary algorithms and the high
computational requirements. Current topics of investigation
focus on levels of abstraction, diversity in the population,

3

Sec. IV - NEW
DIRECTIONS

abstractions
and
representations

evolving
general
learning

incremental
and social
learning

Sec.III
PROGRESS

Evolutionary
algorithms for
EPANNs

evolving
plasticity
rules

fast
learning

evolving
learning
architectures

evolving
deep
learning

evolving
memory

EPANNs in
Evolutionary
Robotics

interaction of
evolution and
learning

Sec. II
I N S P I R AT I O N

Evolved Plastic Artificial
Neural Networks

ev
pr oluti
oc
o
es nar
se
s( y
II.A
)

fitness
objectives

open-ended
evolution
indirect
encodings

catastrophic
forgetting

behavior and
brain functions
learning &
memory

models of pathways

AI in games
machine learning

deep neural learning
reward-based
learning

neuromodulation

C O N T R I B U T I N G

plasticity in artificial
neural networks (II.C)

F I E L D S

indirectly
encoded
plasticity

adaptive behavior

structural plasticity

plasticity in biological
neural networks (II.B)

evolving
neuromodulation

cognitive problems

models of plasiticity
Hebbian plasticity

measuring
progress

control
problems

evolvability
novelty search and
diversity

GPU/
neuromorphic
hardware

A N D

ng )
rni I.D
ea ts (I
l
e
n
tim me
life viron
en

T O P I C S

Figure 1: Organization of the field of EPANNs, reflected in the structure of this paper.
selection criteria, the concepts of evolvability and scalability
(Wagner and Altenberg, 1996; Pigliucci, 2008; Lehman and
Stanley, 2013), the encoding of genetic information through
the genotype-phenotype mapping processes (Wagner and Altenberg, 1996; Hornby et al., 2002), the deception of fitness
objectives and how to avoid them (Lehman and Stanley, 2008;
Stanley and Lehman, 2015). It is also not clear yet which
stepping stones were more challenging for natural evolution
(Roff, 1993; Stanley and Lehman, 2015) in the evolutionary
path to intelligent and complex forms of life. This lack of
knowledge outlines that our understanding of evolutionary
search processes is incomplete, and thus the potential to
exploit these methods is not fully realized. Nevertheless,
artificial evolution provides a tool to search for counterintuitive and innovative solutions that emerge from adaptation
to the environment with minimal or no human intervention.
Effective evolutionary algorithms and their desirable features
for applications to EPANNs are detailed later in Section III-A.
B. Plasticity in biological neural networks
Biological neural networks demonstrate lifetime learning,
from simple reflex adaptation to the acquisition of astonishing
skills such as social behavior and learning to speak one or
more languages. Those skills are acquired by experiencing
stimuli and actions. The fact that experiences guide life-long
learning is an old and intuitive notion. Scientific evidence of
animal learning from experience was documented initially in
the works of behaviorism with scientists such as Thorndike
(1911), Pavlov (1927), Skinner (1938, 1953), and Hull (1943)
who started to test scientifically how experiences cause a
change in behavior, in particular as a result of learning associations and observable behavioral patterns (Staddon, 1983).
This approach means linking behavior to brain mechanisms
and dynamics, an idea initially entertained by Freud (KϕPPE,

1983) and later by other illustrious scientists (Hebb, 1949;
Kandel, 2007). A seminal contribution to link psychology to
physiology came from Hebb (1949), whose ubiquitous principle that neurons that fire together, wire together is relevant
to understanding both low level neural wiring and high level
psychological theories (Doidge, 2007). Much later, a Hebbiancompatible rule that regulates synaptic changes according to
the firing times of the presynaptic and postsynaptic neurons
was observed by Markram et al. (1997) and named SpikeTiming-Dependent Plasticity (STDP).
The seminal work of Kandel and Tauc (1965), and following
studies (Clark and Kandel, 1984), were the first to demonstrate
that changes in the strength of connectivity among neurons
related to behavior learning. Walters and Byrne (1983) showed
that a single neuron can perform associative learning such as
classical conditioning, a class of learning that is observed in
simple neural systems such as that of the Aplysia (Carew et al.,
1981). Plasticity driven by local neural stimuli, i.e. compatible
with the Hebb synapse (Hebb, 1949; Brown et al., 1990), is
responsible not only for fine tuning, but also for building a
working visual system in the cat’s visual cortex (Rauschecker
and Singer, 1981).
Biological plastic neural networks are also capable of
structural plasticity, which creates new pathways among neurons (Lamprecht and LeDoux, 2004; Chklovskii et al., 2004;
Russo et al., 2010): it occurs primarily during development,
but there is evidence that it continues well into adulthood
(Pascual-Leone et al., 2005). Axon growth, known to be regulated by neurotrophic nerve growth factors (Tessier-Lavigne
and Goodman, 1996), was also modeled computationally in
Roberts et al. (2014). Developmental processes and neural
plasticity are often indistinguishable (Kolb, 1989; PascualLeone et al., 2005) because the brain is highly plastic during
development. Neuroscientific advances reviewed in Damasio
(1999); LeDoux (2003); Pascual-Leone et al. (2005); Doidge

4

(2007); Draganski and May (2008) outline the importance of
structural plasticity in learning motor patterns, associations,
and ways of thinking. Both structural and functional plasticity
in biology is essential to acquiring long-lasting new skills, and
for this reason appears to be an important point of inspiration
for EPANNs.
Neuromodulation (Marder and Thirumalai, 2002; Gu, 2002;
Bailey et al., 2000) is a mechanism that mediates neural processes by means of modulatory signals. Modulatory chemicals
such as acetylcholine (ACh), norepinephrine (NE), serotonin
(5-HT) and dopamine (DA) appear to regulate a large variety
of neural functions, from arousal and behavior (Harris-Warrick
and Marder, 1991; Hasselmo and Schnell, 1994; Marder, 1996;
Katz, 1995; Katz and Frost, 1996), to pattern generation (Katz
et al., 1994), to memory consolidation (Kupfermann, 1987;
Hasselmo, 1995; Marder, 1996; Hasselmo, 1999). Learning by
reward in monkeys was linked to dopaminergic activity during
the 1990s with studies by Schultz et al. (1993, 1997); Schultz
(1998). For these reasons, neuromodulation is considered an
essential element in cognitive and behavioral processes, and
has been the topic of a considerable amount of work in
EPANNs (Section III-F).
In summary, plasticity in biological neural networks is
driven by an extraordinarily rich set of dynamics and chemicals. Relevant computational questions for EPANNs include:
(1) How does the brain structure form—driven both by genetic
instructions and neural activity—and acquire functions and
behavior? (2) What are the key plasticity mechanisms from
biology that can be applied to EPANNs? (3) Are memories,
skills, and behaviors stored in plastic synaptic connections,
in patterns of activities, or a combination of both? Whilst
neuroscience continues to provide inspiration and insight into
plasticity in biological brains, EPANNs serve the complementary objective of seeking, implementing, and verifying
designs of bio-inspired methods for adaptation, learning, and
intelligent behavior.
C. Plasticity in artificial neural networks
In EPANN experiments, evolution can be seen as a metalearning process. Established learning rules are often used as
ingredients that evolution uses to search for good parameter
configurations, efficient combinations of rules and network
topologies, new functions representing novel learning rules,
etc. EPANN experiments are suited to include the largest
possible variety of rules because of (1) the variety of possible tasks in a simulated behavioral experiment and (2) the
flexibility of evolution to combine rules with no assumptions
about their dynamics. The following gives a snapshot of the
extent and scope of various learning algorithms for ANN that
can effectively be building blocks of EPANNs.
In supervised learning, backpropagation is the most popular
learning rule used to train both shallow and deep networks
(Rumelhart et al., 1988; Widrow and Lehr, 1990; LeCun et al.,
2015) for classification or regression. Unsupervised learning
is implemented in neural networks with self-organizing maps
(SOM) (Kohonen, 1982, 1990), auto-encoders (Bourlard and
Kamp, 1988), restricted Boltzmann machines (RBM) (Hinton

and Salakhutdinov, 2006), Hebbian plasticity (Hebb, 1949;
Gerstner and Kistler, 2002a; Cooper, 2005), and various
combinations of the above (Willshaw and Dayan, 1990).
Hebbian rules, given their biological plausibility and unsupervised learning, are a particularly important inspirational
principle. Variations have been proposed to include, e.g., terms
to achieve stability (Oja, 1982; Bienenstock et al., 1982)
and various constraints (Miller and Mackay, 1994), or more
advanced update dynamics such as dual weights for fast and
slow decay (Levy and Bairaktaris, 1995; Hinton and Plaut,
1987; Bullinaria, 2009a; Soltoggio, 2015). The final learning
dynamics of those rules depend often on the topology of the
network and on the parameters of the rule. New Hebbian
rules have been recently proposed to minimize defined cost
functions (Pehlevan et al., 2015; Bahroun et al., 2017).
Often, behavior learning requires a measure of the quality
of such behaviors, implemented as positive or negative values
that represent the consequences of stimulus-action sequences.
This type of learning, also referred to as trial and error, or
operant reward learning in animal learning, was formalized
mathematically as Reinforcement Learning (RL) (Sutton and
Barto, 1998). RL in neural networks (Pennartz, 1997) makes
use of robotic or agent-environment problems with a reward
signal that affects plasticity in the network (Lin, 1993; Niv
et al., 2002; Soltoggio and Stanley, 2012; Auerbach et al.,
2014; Pugh et al., 2014). Neuromodulated plasticity (Fellous
and Linster, 1998) is often used to implement reward-learning
in neural networks (reviewed later in Section III-F). Such
a modulation of signals, or gated learning (Abbott, 1990),
allows for the regulation and learning of behaviors as shown
in numerous computational models (Baxter et al., 1999; Suri
et al., 2001; Birmingham, 2001; Alexander and Sporns, 2002;
Doya, 2002; Fujii et al., 2002; Suri, 2002; Ziemke and Thieme,
2002; Sporns and Alexander, 2003; Krichmar, 2008).
Plastic neural models are also used to demonstrate how
behavior can emerge from a particular circuitry modeled
after biological brains. Computational models of, e.g., the
basal ganglia and modulatory systems may propose plasticity mechanisms and aim to demonstrate the computational
relations among various nuclei, pathways, and learning processes (Krichmar, 2008; Vitay and Hamker, 2010; Schroll and
Hamker, 2015).
Finally, plasticity rules for spiking neural networks (Maass
and Bishop, 2001) aim to demonstrate unique learning mechanisms that emerge from spiking dynamics (Markram et al.,
1997; Izhikevich, 2006, 2007), as well as model biological
synaptic plasticity (Gerstner and Kistler, 2002b).
Plasticity in neural networks, when continuously active, was
also observed to cause catastrophic forgetting (Robins, 1995).
If learning occurs continuously, new information or skills have
the potential to overwrite previously acquired information or
skills (Abraham and Robins, 2005; Finnie and Nader, 2012).
In conclusion, plasticity is central to most approaches to
ANNs, but in the majority of cases, a careful matching and
engineering of rules, architectures and problems are necessary,
requiring considerable design effort. The variety of algorithms
also reflects the variety of problems and solutions. EPANNs
offer a unique testing tool to assess the effectiveness and

5

suitability of different models, or their combination, in a
variety of different scenarios.

process, neural plastic models, and a learning environment:
a simplified graphical representation of an EPANN setup is
given in Fig. 2.

D. Lifelong learning environments
Evolution works on organisms within an environment to
exploit its resources, find survival solutions, and adapt to
changes (Darwin, 1859). However, what makes an environment conducive to the evolution of learning and intelligence
is not clear. In particular, what are the challenges faced by
learning organisms in the natural world, and how can those be
abstracted and ported to a simulated environment?
In the early phases of AI, logic and reasoning were thought
to be key to intelligence (Cervier, 1993), so symbolic inputoutput mappings were employed as tests. Soon it became
evident that intelligence is not only symbol manipulation, but
resides also in subsymbolic problem solving abilities emerging
from the interaction of brain, body, and environment (Steels,
1993; Sims, 1994), so more complex simulators of real-life
environments were developed. A further step in complexity
is represented by high-level planning and strategies required,
e.g., when applying AI to games (Allis et al., 1994; Millington
and Funge, 2016) or articulated robotic tasks. Planning and
decision making with high bandwidth sensory-motor information flow such as those required for humanoid robots or selfdriving vehicles are current benchmarks of intelligent systems.
Finally, environments in which affective dynamics and feelings
play a role are recognized as important for human well
being (De Botton, 2016; Lee and Narayanan, 2005). Those
intelligence-testing environments are effectively the “worlds”
in which EPANNs live and evolve, and thus largely shape the
design process.
Such different testing environments have very different
features, dynamics, and goals. In some cases, a precise target
behavior exists and is known, and an error between desired
and actual behavior can be used in supervised learning. Other
times, the environment provides rewards, but the behaviors
that lead to rewards are unknown. In such cases, the objective
is to search for behavioral policies that lead to collecting
rewards. In yet other cases, it is useful to find relationships and
regularities in the environment to construct complex behaviors
and accomplish particular tasks, which can either lead to
reward, exploit resources, or guarantee survival. Finally, recent
advances in the theory of evolutionary computation (Lehman
and Stanley, 2011; Stanley and Lehman, 2015) suggest that is
not always the fittest, but at times is the novel individual or
behavior that can exploit environmental niches, thus leading
to creative evolutionary processes as those observed in nature.
Such an increasingly large range of testing environments and
multi-skill integration also calls for new evaluation metrics
(see Section IV-H).
III. P ROGRESS ON EVOLVING ARTIFICIAL PLASTIC
NEURAL NETWORKS

After outlining important features of evolutionary algorithms that operate with EPANNs (Section III-A), this section
proceeds to review studies that have evolved plastic neural
networks. Those studies generally include an evolutionary

A. Evolutionary algorithms for EPANNs
As opposed to parameter optimization (Bäck and Schwefel,
1993) in which search spaces are often of fixed dimension
and static, EPANNs evolve in dynamic search spaces in
which lifetime learning further increases the complexity of the
evolutionary search. Evolutionary algorithms (Holland, 1975;
Michalewicz, 1994) for EPANNs often require additional
advanced features to cope with the challenges of the evolution
of learning and open evolutionary design (Bentley, 1999).
Desirable properties of EA for EPANNs are listed in the
following.
1) Variable genotype length and growing complexity: In
many learning problems, the complexity and the characteristics
of the solution required to solve the problem are not known
in advance. Therefore, a desirable property of the EA is
that of increasing the length of the genotype, and thus the
information contained in it, as evolution discovers increasingly
more complex strategies and solutions.
2) Indirect genotype to phenotype encoding: In nature, phenotypes are expressions of a more compact representation, the
genetic code. Similarly, evolutionary algorithms may represent
genetic information in a compact form, which is then mapped
to a larger phenotype.
3) Expressing regularities, repetitions, and patterns: Indirect encodings are beneficial when they can use one set
of instructions in the genotype to generate more parts in
the phenotype. This may involve expressing regularities like
symmetry (e.g. symmetrical neural architectures), repetition
(e.g. neural modules), repetition with variation (similar neural
modules), and patterns (e.g. motifs in the neural architecture).
4) Effective exploration via mutation and recombination:
Genetic mutation and sexual reproduction in nature allow
for the expression of a variety of phenotypes and for the
exploration of new solutions, but seldom lead to highly unfit
individuals (Ay et al., 2007). Similarly, EAs for EPANNs
need to be able to mutate and recombine genomes without
destroying the essential properties of the solutions. As opposed
to other heuristic searches, EAs use recombination to generate
new solutions from two parents: how to effectively recombine
genetic information from two EPANN is still an open question.
5) Genetic encoding of plasticity rules: Just as neural networks need a genetic encoding to be evolved, so do plasticity
rules. EPANN algorithms require the integration of such a rule
in the genome. The encoding may be restricted to a simple
parameter search, but the field is moving progressively towards
the evolution of arbitrary and general plasticity rules. Plasticity
may also be applied to all or parts of the network, thus effectively implementing the evolution of learning architectures.
6) Diversity, survival criteria, and low selection pressure:
The variety of solutions in nature seems to suggest that
diversity is a key aspect of natural evolution. EAs for EPANNs
are likely to perform better when they can maintain diversity
in the population, both at the genotype and phenotype levels.

6

evolution via
EPANN-specific
algorithms

interaction of
evolution and
learning

learning
parameters
variable genotype
length
effective mutation/
(a) Neural
recombination

direct
mapping

Network

indirect

lifetime
learning

plasticity
functions

plastic
architectures:
acquired skills

(b) Sensors

modulatory

Figure 1: Maze Navigating Robot.
The artificial neural
network
neurons
mapping:

that representations
controls the maze navigatingregularities
robot is shown in (a). The layout
genetic
the sensors
is shown in (b). repetitions
Each arrow outside offixed
the robot’s
ofofneural
systems
patterns
body in (b) is a rangefinder sensor
that indicatesarchitectures
the distance to
instincts
&
the closest
obstacle in that direction. The robot has
four pie-slice
genetic
representations
learning
sensors
thatrules
act as a compass towards the goal, activating
when a
of
plasticity
methods

output

sensory
motor
experiences

(a) Medium Map

(b) Hard Map

navigation
In both
maps, the large circle
(e.g. mazes)
represents the starting position of the robot and the small circle
represents the goal. Cul-de-sacs in both maps that lead toward the
behavioral
foraging
goal create the potential for
deception.
diversity

Figure 2:

reward
Maze Navigation
Maps.
learning

control

Fitness-based NEAT, which will be compared to novelty
adaptation
/
search, requires
a fitness
function to reward
maze-navigating
survival/
input
self regulation
speciation
reproduction
robots. Because the objective
is to reach the goal, the fitness
modules
f is defined as the distance from the robot to the goal at the
expanding the neural networks in NEAT) to create novel bepool of genes
mapping
EPANNend of an evaluation: f = benvironment
dg , where bf is a constant
f
haviors as the simpler variants are exhausted.
bias and dg is the distance from the robot to the goal. Given
fitness / objective a maze with no deceptive obstacles, this fitness function deExperiment
fines a monotonic gradient for search to follow. The constant
A good domain for testing novelty search should have a debf ensures
will have (right)
positive allow
fitness. for plastic
Figure 2: Main ceptive
elements
of
an
EPANN
setup
in
which
simulated
evolution
(left) all
andindividuals
an environment
fitness landscape. In such a domain, the search alNEAT
with noveltyThe
search,
on the may
other include
hand, requires
networks to evolve
through
generations
and
learn
within
a
lifetime
in
the
environment.
network
diverse
gorithm following the fitness gradient may perform worse
a novelty metric to distinguish between maze-navigating
an algorithm
following novelty
gradientsneurons,
because novelements, e.g., than
modules
or architectures,
modulatory
inputs/outputs,
learning
parameters,
and
other
structures
that
robots. Defining the novelty metric requires careful conenable learning.elty cannot be deceived; it ignores fitness entirely. A comsideration because it biases the search in a fundamentally
pelling, easily-visualized domain with this property is a twodifferent way than the fitness function. The novelty metdimensional maze navigation task. A reasonable fitness
ric determines the behavior-space through which search will
function for such a domain is how close the maze navigaNiche exploration and behavioral diversity could also play a and III-A3).
Novelty
search (Lehman
andofStanley,
2008,
2011)
proceed.
It is important
that the type
behaviors
that one
tor is to the goal at the end of the evaluation. Thus, dead
hopes to distinguish
are recognized
bysurvival
the metric.of the fittest
key role for creative
design
processes.
Low
selection
pressure
was
introduced
as
an
alternative
to
the
ends that lead close to the goal are local optima to which
Thus, for the maze domain, the behavior of a navigator is
might be crucialantoobjective-based
evolve learning
in deceptive
environments
algorithm
may converge,
which makesasa a selection mechanism (III-A6). Initially, these algorithms
defined
as itstoending
position.
novelty metric
is then the
(see Section III-E).
were not devised
evolve
plasticThe
networks,
but rather
fixed
good model for deceptive problems in general.
Euclidean distance between the ending positions of two inThis paper’s experiments utilize NEAT, which has been
networks in which the final synaptic weights were encoded
dividuals. For example, two robots stuck in the same corner
7) Open-ended
evolution:
evolutionary
experiments
proven
in many Simple
control tasks
[27, 29], including
maze naviin the genome.
To operate
appear similar,
while onewith
robotEPANNs,
that simplythese
sits atalgorithms
the start
seem to quickly
optimize
parameters
and
reach
a
fitness
gation [26], the domain of the experiments in this paper. need to integrate additional genotypical instructions to evolve
position
looks
very
different
from
one
that
reaches
the goal,
plateau without further
improvement.
The maze
domain works Open-ended
as follows. A evolution
robot controlled
plasticitythough
rulesthey
(III-A5).
are both equally viable to the novelty metric.
an ANN must
from aindefinitely
starting pointmore
to an end
seeks to set up by
experiments
thatnavigate
can evolve
This
novelty
metric
rewards
the robot
for endingneuroevoluin a place
By
combining
these
features
(III-A1
- III-A7),
pointgiven
in a fixed
time. The
task is complicated
by(Taycul-de-sacs
complex solutions
sufficient
computational
power
where none
have ended
the method
of traversal
igtion of plastic
networks
aimsbefore;
to search
extremely
largeissearch
that
prevent
a
direct
route
and
that
create
local
optima
in
the
lor et al., 2016).
nored.
This
measure
reflects
that
what
is
important
is
reachspaces in fundamentally different and more creative ways
fitness landscape. The robot (figure 1) has six rangefinding a certain location (i.e. the goal) rather than the method
Many evolutionary
includetoone
moreobstacle
of those
ers that algorithms
indicate the distance
the or
nearest
and four
than traditional
heuristic searches of parameters and hyperof locomotion. Thus, although the novelty metric has no
pie-slice
radar sensors
the goal is within
desirable properties
(Fogel,
2006). that
Duefire
towhen
the complexity
of the
parameters.
knowledge of the final goal, a solution that reaches the goal
twoneuroevolution
effectors result inwas
forcesthe
that reneural network pie-slice.
design, The
the robot’s
field of
will appear novel. Furthermore, the comparison between
spectively
turn
and
propel
the
robot.
This
setup
is
similar
to
first to explore most of those extensions of standard evofitness-based
and rules
novelty-based search is fair because both
B. Evolving
plasticity
the successful maze navigating robots in NERO [26].
lutionary algorithms.
Popular algorithms include early work
scores
are
computed
only based on the distance of the final
Two maps are designed to compare the performance of Early experiments evolved
parameters
of learning rules
of Angeline et NEAT
al. (1994)
Yao andsearch
Liu (1997)
to evolve
position of the robot fromthe
other
points.
with and
fitness-based
and NEAT
with novelty
for
fixed
or
hand-designed
ANN
architectures.
Learning
rules
Finally, to confirm that novelty search is indeed
not anyfixed weights (i.e.
weights
do not2a)
change
while the
agent
search.
The that
first (figure
has deceptive
dead
ends that
are
functions
that
change
the
connection
weight
w
between
thing like random search, NEAT is also tested with a raninteracts with its
environment)
thegoal.
topology
of arbitrary
lead
the robot closeand
to the
To achieve
a higher fitness
dom fitness
assigned
to every
individualasregardless of pertwo neurons,
and are
generally
expressed
local optimum
provided
by a dead
end, the robot
neural networks,than
e.g.,therecurrent
(addressing
III-A1
and III-A4).
formance,
which
means
that
selection
is random. If the maze
travel part of the
way through(NEAT)
a more difficult
path that
Neuroevolution must
of Augmenting
Topologies
(Stanley
∆w of
= evaluations
f (x, θ) ,is recorded.
(1)
is solved, the number
requires a weaving motion. The second maze (figure 2b)
line from the goal to the center of the robot falls within the pieslice. The solid arrow indicates the robot’s heading.

and Miikkulainen, 2002) leverages on three main aspects: a
provides a more deceptive fitness landscape that requires the
is a vector of Parameters
neural signals and θ is a vector of fixed
recombination operator
intended to preserve network func- where xExperimental
search algorithm to explore areas of significantly lower fitparameters. The incoming connection weights w to a neuron
tion (addressingness
III-A4);
speciation
(addressing
III-A6);
and
before finding the global optimum (which is a network
Because NEAT with novelty search differs from origdetermine
activation value
evolution from that
small
to larger
networks (addressing III-A1). i are used
reaches
the goal).
inaltoNEAT
only the
in substituting
a novelty metric for
X

Analog Genetic Encoding (AGE) (Mattiussi and Floreano,
xi = σ
(wji · xj ) ,
(2)
2007) is a method for indirect genotype to phenotype mapping
that can be used in combination with evolution to design where xj are the activation values of presynaptic neurons that
arbitrary network topologies (III-A1 and III-A2). HyperNEAT connect to neuron i with the weights wji , and σ is a nonlinear
(Stanley et al., 2009) is an indirect representation method that function such as the sigmoid or the hyperbolic tangent. x may
combines the NEAT algorithm with compositional patterns provide local signals such as pre and postsynaptic activities,
producing networks (CPPN) (Stanley, 2007) (III-A1, III-A2 the value of the weight w, and modulatory or error signals.

7

Bengio et al. (1990, 1992) proposed the optimization of the
parameters θ of generic learning rules with gradient descent,
simulated annealing, and evolutionary search for problems
such as conditioning, boolean function mapping, and classification. Those studies are also among the first to include a
modulatory term in the learning rules. The optimization was
shown to improve the performance for those different tasks
with respect to manual parameter settings. Chalmers (1990)
evolved a learning rule that applied to every connection and
had a teaching signal. He discovered that, in 20% of the
evolutionary runs, the algorithm evolved the well-known delta
rule, or Widrow-Hoff rule (Widrow et al., 1960), used in backpropagation, thereby demonstrating the validity of evolution
as an autonomous tool to discover learning. Fontanari and
Meir (1991) used the same approach of Chalmers (1990) but
constrained the weights to binary values. Also in this case,
evolution autonomously rediscovered a hand-designed rule, the
directed drift rule by Venkatesh (1993). They also observed
that the performance on new tasks was better when the network
evolved on a larger set of tasks, possibly encouraging the
evolution of more general learning strategies.
With back-propagation of errors (Widrow and Lehr, 1990),
the input vector x of Eq. 1 requires an error signal between
each input/output pair. Rules that use only local signals have
been a more popular choice for EPANNs, though this is
already changing with the rise in effectiveness of deep learning
using back-propagation and related methods. In the simplest
form, the product of presynaptic (xj ) and postsynaptic (xi )
activities, and a learning rate η
∆w = η · xj · xi

.

(3)

is known as Hebbian plasticity (Hebb, 1949; Cooper, 2005).
More generally, any function as in Eq. 1 that uses only local
signals is considered a local plasticity rule for unsupervised
learning. Baxter (1992) evolved a network that applied the basic Hebbian rule in Eq. 3 to a subset of weights (determined by
evolution) to learn four boolean functions of one variable. The
network, called Local Binary Neural Net (LBNN), evolved
to change its weights to one of two possible values (±1),
or have fixed weights. The experiment proved that learning
can evolve when rules are optimized and applied to individual
connections.
Nolfi and Parisi (1993) evolved networks with “autoteaching” inputs, which could then provide an error signal for
the network to adjust weights during lifetime. The implication
is that error-signals do not always need to be hand-designed
but can be discovered by evolution to fit a particular problem.
A set of eight different local rules was used in Rolls and
Stringer (2000) to investigate the evolution of rules in combination with the number of synaptic connections for each neuron, different neuron classes, and other network parameters.
They found that evolution was effective in selecting from large
parameter sets to solve simple linear problems. In Maniadakis
and Trahanias (2006), co-evolution was used to evolve agents
(each being a network) that could use ten different types of
Hebbian-like learning rules for simple navigation tasks: the
authors reported that, despite the increase in the search space,
using many different learning rules results in better perfor-

mance but, understandably, a more difficult analysis of the
evolved systems. Meng et al. (2011) evolved a gene regulatory
network that in turn determined the learning parameters of
the BCM rule (Bienenstock et al., 1982), showing promising
performance in time series classification and other supervised
tasks.
One general finding from these studies is that evolution
operates well within large search spaces, particularly when
a large set of evolvable rules is used.
C. Evolving learning architectures
The interdependency of learning rules and neural architectures led to experiments in which evolution had more
freedom on the network’s design, including the architecture.
The evolution of architectures in ANNs may involve searching
an optimal number of hidden neurons, the number of layers
in a network, particular topologies or modules, the type of
connectivity, and other properties of the network architecture.
In EPANNs, evolving learning architectures implies more
specifically to discover a combination of architectures and
learning rules whose synergetic matching enables particular learning dynamics. As opposed to biological networks,
EPANNs do not have the neurophysiological constraints, e.g.,
short neural connections, sparsity, brain size limits, etc., that
impose limitations on the natural evolution of biological networks. Thus, biologically implausible artificial systems may
nevertheless be evolved in-silico (Bullinaria, 2007b, 2009b).
One seminal early study by Happel and Murre (1994) proposed the evolutionary design of modular neural networks in
which modules could perform unsupervised learning, and the
intermodule connectivity was shaped by Hebbian rules. The
finding was that the use of evolution led to enhanced learning
and better generalization capabilities in comparison to handdesigned networks. In Arifovic and Gencay (2001), the authors
used evolution to optimize the number of inputs and hidden
nodes, and allowed connections in a feedforward neural network to be trained with backpropagation. Abraham (2004) proposed a method called Meta-Learning Evolutionary Artificial
Neural Networks (MLEANN) in which evolution searches for
initial weights, neural architectures and transfer functions for a
range of supervised learning problems to be solved by evolved
networks. The evolved networks were tested in time series prediction and compared with manually designed networks. The
analysis showed that evolution consistently found networks
with better performance than the hand-designed structures.
Khan et al. (2008) proposed an evolutionary developmental
system that created an architecture that adapted with learning:
the network had a dynamic morphology in which neurons
could be inserted or deleted, and synaptic connections formed
and changed in response to stimuli. The networks were evolved
with Cartesian genetic programming and appeared to improve
their performance while playing checkers over the generations.
Downing (2007) looked at different computational models of
neurogenesis to evolve learning architectures. The proposed
evolutionary developmental system focused in particular on
abstraction levels and principles such as Neural Darwinism
(Edelman and Tononi, 2000). A combination of evolution of

8

recurrent networks with a linear learner in the output was
proposed in Schmidhuber et al. (2007), showing that the
evolved RNNs were more compact and resulted in better
learning than randomly initialized echo state networks (Jaeger
and Haas, 2004). In Khan et al. (2011b,a); Khan and Miller
(2014), the authors introduced a large number of bio-inspired
mechanisms to evolve networks with rich learning dynamics.
The idea was to use evolution to design a network that was
capable of advanced plasticity such as dendrite branch and
axon growth and shrinkage, neuron insertion and destruction,
and many others. The system was tested on the Wumpus World
(Russell and Norvig, 2013), a fairly simple problem with no
online learning required, but the purpose was to show that
evolution can design working control networks even within a
large search space of developmental dynamics.
In summary, learning mechanisms and neural architectures
are strongly interdependent, but a large set of available dynamics seem to facilitate the evolution of learning. Thus, EPANNs
become more effective precisely when manual network design
becomes less practical because of complexity and rich dynamics.
D. EPANNs in Evolutionary Robotics
Evolutionary robotics (ER) (Cliff et al., 1993; Floreano and
Mondada, 1994, 1996; Urzelai and Floreano, 2000; Floreano
and Nolfi, 2004) contributed particularly to the development
of EPANNs, providing a testbed for applied controllers in
robotics. Although ER had no specific assumptions on neural systems or plasticity (Smith, 2002), robotics experiments
suggested that neural control structures evolved with fixed
weights perform less well than those evolved with plastic
weights (Nolfi and Parisi, 1996; Floreano and Urzelai, 2001b).
Floreano and Urzelai (2001a) reported that networks evolved
faster when synaptic plasticity and neural architectures were
evolved simultaneously. In particular, plastic networks were
shown to adapt better in the transition from simulation to real
robots. The better simulation-to-hardware transition, and the
increased adaptability in changing ER environments, appeared
intuitive and supported by evidence (Nolfi and Parisi, 1996).
However, the precise nature and magnitude of the changes
from simulation to hardware is not always easy to quantify:
those studies do not clearly outline the precise principles, e.g.,
better or adaptive feedback control, minimization principles,
etc., that are discovered by evolution with plasticity to produce
those advantages. In fact, the behavioral changes required to
switch behaviors in simple ER experiments can also take place
with non-plastic recurrent neural networks because inputs can
switch the network’s activity to different attractors. These
dynamics were observed in particular in recurrent networks in
which neurons are modeled as leaky integrators with different
time constants. Such networks were called continuous time
recurrent neural networks (CTRNN) (Beer and Gallagher,
1992; Yamauchi and Beer, 1994). A study in 2003 observed
similar performance in learning when comparing plastic and
non-plastic CTRNNs (Stanley and Miikkulainen, 2003b). That
study indicates that the learning of plastic networks as described in Blynel and Floreano (2002, 2003) was at that point

still a proof-of-concept rather than a superior learning tool
because networks with fixed weights could achieve similar
performance. Nevertheless, ER maintained a focus on plasticity as demonstrated, e.g., in The Cyber Rodent Project (Doya
and Uchibe, 2005) that investigated the evolution of learning
by seeking to implement a number of features such as (1)
evolution of neural controllers, (2) learning of foraging and
mating behaviors, (3) evolution of learning architectures and
meta-parameters, (4) simultaneous learning of multiple agents
in a body, and (5) learning and evolution in a self-sustained
colony. Plasticity in the form of modulated neural activation
was used in Husbands et al. (1998) and Smith et al. (2002) with
a network that adapts its activation functions according to the
diffusion of a simulated gas spreading to the substrate of the
network. Although the robotic visual discrimination tasks did
not involve lifetime learning, the plastic networks appeared to
evolve faster than a network evolved with fixed activation functions. Similar conclusions were reached in Di Paolo (2003) and
Federici (2005). Di Paolo (2002, 2003) evolved networks with
STDP for a wheeled robot to perform positive and negative
phototaxis, depending on a conditioned stimulus, and observed
that networks with fixed weights could learn but had inferior
performance with respect to plastic networks. Federici (2005)
evolved plastic networks with STDP and an indirect encoding,
showing that plasticity helped performance even if adaptivity
was not required. Stability and evolvability of simple robotic
controllers were investigated in Hoinville et al. (2011) who
focused on EPANNs with homeostatic mechanisms.
Experiments in ER in the 1990s and early 2000s revealed
the extent, complexity, and multitude of ideas behind the
evolutionary design of learning neuro-robotics controllers.
They generally indicate that plasticity helps evolution under
a variety of conditions, even when lifetime learning is not
required, thereby promoting further interest in more specific
topics. Among those are the interaction of evolution and
learning, the evolution of neuromodulation, and the evolution
of indirectly encoded plasticity, as described in the following.
E. Interaction of evolution and learning
EPANN algorithms are influenced by the complex interaction of evolution and learning. Such interaction dynamics are
present in any system, i.e., beyond EPANNs, that seeks the
evolution of learning. Lifetime learning was first suggested to
have an impact on accelerating evolution by Baldwin (1896).
The studies in Smith (1986); Hinton and Nowlan (1987);
Boers et al. (1995); Mayley (1996); Bullinaria (2001) demonstrated the Baldwin effect with computational simulations. For
example, Mayley (1997) investigated the effects of learning
on the rate of evolution, showing that a learning process
during an individual lifetime, although costly, can improve
the speed of evolution of the species. In general, learning
can be classified in (1) learning of invariant environmental
features, typically affected by the Baldwin effect, and (2)
learning variable environmental features. A large body of work
is dedicated to study the interaction and dynamics of all three
POE dimensions, e.g. Paenke (2008); Paenke et al. (2009).
Paenke et al. (2007) investigated the influence of plasticity

9

Fitness/Reward

234
A learning strategy
is discovered by an
220 EPANN
210
200

A navigation strategy
is first evolved without
reward-learning

190
178
0

1000

2000

2500

Generations

Figure 3: Discovery of a learning strategy during evolution.
After approximately 1 000 generations, a reward signal is used
by one network to learn where is the changing location of a
stochastic reward. Figure adapted from Soltoggio et al. (2007).
This experiment suggests that the evolution of learning may
proceed through milestones without obvious gradients in the
search.
and learning on evolution under directional selection; Mery
and Burns (2010) demonstrate that behavioral plasticity affects
evolution.
In EPANNs, similarly complex interaction dynamics between evolution and learning were also observed (Nolfi and
Floreano, 1999). Stone (2007) showed that distributed neural
representations accelerate the evolution of adaptive behavior
because learning part of a skill induced the automatic acquisition of other skill components. One study (Soltoggio et al.,
2007) suggested that evolution discovers learning by discrete
stepping stones: when evolution casually discovers a weak
mechanism of learning, it is sufficient to create an evolutionary
advantage, so the neural mechanism is subsequently evolved
quickly to perfect learning: Fig. 3 shows a sudden jump in the
fitness when the agent suddenly evolves a learning strategy:
such jumps in fitness graphs are common in evolutionary
experiments in which learning is discovered from scratch,
rather than optimized, and were observed as early as in
Fontanari and Meir (1991).
When an environment changes over time, the frequency of
those changes plays a role. With timescales comparable to a
lifetime, evolution may lead to phenotypic plasticity, which is
the capacity for a genotype to express different phenotypes in
response to different environmental conditions (Lalejini and
Ofria, 2016). The frequency of environmental changes was
observed experimentally in plastic neural networks to affect
the evolution of learning (Ellefsen, 2014), revealing a complex
relationship between environmental variability and evolved
learning.
The evolution of neural learning was shown to be inherently
deceptive in Risi et al. (2010); Lehman and Miikkulainen
(2014). In Risi et al. (2009, 2010), EPANN-controlled simulated robots, evolved in a discrete T-Maze domain, revealed
that the evolutionary stepping stones towards adaptive behaviors are often not rewarded by objective-based performance measures. In other words, the necessary evolutionary
stepping stones towards achieving adaptive behavior receive

a lower fitness score than more brittle solutions with low
learning but effective behaviors. A solution to this problem
was devised in Risi et al. (2010, 2009), in which novelty
search (Lehman and Stanley, 2008, 2011) was adopted as a
substitute for performance in the fitness objective with the
aim of finding novel behaviors. Novelty search was observed
to perform significantly better in the T-Maze domain. Lehman
and Miikkulainen (2014) later showed that novelty search can
encourage the evolution of more adaptive behaviors across a
variety of different variations of the T-Maze learning tasks.
As a consequence, novelty search contributed to a philosophical change by questioning the centrality of objectivedriven search in current evolutionary algorithms (Stanley
and Lehman, 2015). By rewarding novel behaviors, novelty
search validates the importance of exploration or curiosity,
previously proposed in Schmidhuber (1991, 2006), also from
an evolutionary viewpoint. With the aim of validating the
same hypothesis, Soltoggio and Jones (2009) devised a simple
EPANN experiment in which exploration was more advantageous than exploitation in the absence of reward learning; to
do this, the reward at a particular location depleted itself if
continuously visited, so that changing location at random in
a T-maze became beneficial. Evolution discovered exploratory
behavior before discovering reward-learning, which in turn,
and surprisingly, led to an earlier evolution of reward-based
learning. Counterintuitively, this experiment suggests that a
stepping stone to evolve reward-based learning is to encourage
non-reward based exploration.
The seminal work in Bullinaria (2003, 2007a, 2009c) proposes the more general hypothesis that learning requires the
evolution of long periods of parental protection and late onset
of maturity. Similarly, Ellefsen (2013b,a) investigates sensitive
and critical periods of learning in evolved neural networks.
This fascinating hypothesis has wider implications for experiments with EPANNs, and more generally for machine learning
and AI. It is therefore foreseeable that future EPANNs will
have a protected childhood during which parental guidance
may be provided (Clutton-Brock, 1991; Klug and Bonsall,
2010; Eskridge and Hougen, 2012).
F. Evolving neuromodulation
Growing neuroscientific evidence on the role of neuromodulation (previously outlined in Section II-B) inspired the design
of experiments with neuromodulatory signals to evolve control
behavior and learning strategies. One particular case is when
neuromodulation gates plasticity. Eq. 1 can be rewritten as as
∆w = m · f (x, θ) ,

(4)

to emphasize the role of m, a modulatory signal used as
a multiplicative factor that can enhance or reduce plasticity
(Abbott, 1990). A network may produce many independent
modulatory signals m targeting different neurons or areas of
the network. Thus, modulation can vary in space and time.
Modulation may also affect other aspects of the network
dynamics, e.g., modulating activations rather than plasticity
(Krichmar, 2008). Graphically, modulation can be represented
as a different type of signal affecting various properties of the
synaptic connections of an afferent neuron i (Fig. 4).

10

wki

m
synthesis of
control signals

modulated
network

standard
neuron
j

xj
plastic
connection

equivalent but different control structures. Fig. 4 also suggests
that modulatory networks require evolution to find two essential topological structures: what signals or combination of
signals trigger modulation, and what neurons are to be targeted
by modulatory signals. In other words, a balance between
fixed and plastic architectures, or selective plasticity (DARPAL2M, 2017), is an intrinsically emergent property of evolved
modulated networks.

modulatory
connection

modulatory
neuron
k

synthesis of
learning signals

modulatory
signal
standard
neuron
i

xi

wji

Figure 4: A modulatory neuron gates plasticity of the synapses
that connect to the postsynaptic neuron. The learning is local,
but a learning signal can be created by one part of the network
and used to regulate learning elsewhere.
Evolutionary search was used to find the parameters of
a neuromodulated Hebbian learning rule in a reward-based
armed-bandit problem in Niv et al. (2002). The same problem
was used later in Soltoggio et al. (2007) to evolve arbitrary
learning architectures with a bio-inspired gene representation
method called Analog Genetic Encoding (AGE) (Mattiussi and
Floreano, 2007). In that study, evolution was used to search
both modulatory topologies and parameters of a particular
form of Eq. 4:
∆w = m · (Axi xj + Bxj + Cxi + D) ,

(5)

where the parameters A to D determined the influence of four
factors in the rule: a multiplicative Hebbian term A, a presynaptic term B, a postsynaptic term C, and pure modulatory,
or heterosynaptic, term D. Such a rule is not dissimilar from
those presented in previous studies (see Section 3.2). However,
when used in combination with modulation and a search
for network topologies, evolution seems to be particularly
effective at solving reward-based problems. Kondo (2007)
proposed an evolutionary design and behavior analysis of
neuromodulatory neural networks for mobile robot control,
validating the potential of the method.
Soltoggio et al. (2008) tested the hypothesis of whether
modulatory dynamics held an evolutionary advantage in Tmaze environments with changing reward locations. In their
algorithm, modulatory neurons were freely inserted or deleted
by random mutations, effectively allowing the evolutionary
selection mechanism to autonomously pick those networks
with advantageous computational components. After evolution, the best performing networks had modulatory neurons
regulating learning, and evolved faster than a control evolutionary experiment that could not employ modulatory neurons.
Modulatory neurons were maintained in the networks in a second phase of the experiment when genetic operators allowed
for the deletion of such neurons but not for their insertion,
thus demonstrating their essential function. In another study,
Soltoggio (2008b) suggested that evolved modulatory topologies may be essential to separate the learning circuity from
the input-output controller, effectively rediscovering an actorcritic architecture, and shortening the input-output pathways
which sped up decision processes. Soltoggio (2008a) showed
that the learning dynamics are affected by tight coupling
between rules and architectures in a search space with many

A number of further studies on the evolution of neuromodulatory dynamics confirmed the evolutionary advantages in
learning scenarios (Soltoggio, 2008c). Silva et al. (2012a) used
simulations of 2-wheel robots performing a dynamic concurrent foraging task, in which scattered food items periodically
changed their nutritive value or become poisonous, similarly to
the setup in Soltoggio and Stanley (2012). The results showed
that when neuromodulation was enabled, learning evolved
faster than when neuromodulation was not enabled, also with
multi-robot distributed systems (Silva et al., 2012b). Nogueira
et al. (2013) also reported evolutionary advantages in foraging
behavior of an autonomous virtual robot when equipped with
neuromodulated plasticity. Harrington et al. (2013) demonstrated how evolved neuromodulation applied to a gene regulatory network consistently generalized better than agents
trained with fixed parameter settings. Interestingly, Arnold
et al. (2013b) showed that neuromodulatory architectures
provided an evolutionary advantage also in reinforcement-free
environments, validating the hypothesis that plastic modulated
networks have higher evolvability in a large variety of tasks.
The evolution of social representations in neural networks
was shown to be facilitated by neuromodulatory dynamics in
Arnold et al. (2013a). An artificial life simulation environment
called Polyword (Yoder and Yaeger, 2014) helped to assess the
advantage of neuromodulated plasticity in various scenarios.
The authors found that neuromodulation may be able to
enhance or diminish foraging performance in a competitive,
dynamic environment.
Neuromodulation was evolved in Ellefsen et al. (2015) in
combination with modularity to address the problem of catastrophic forgetting. In Gustafsson (2016), networks evolved
with AGE (Mattiussi and Floreano, 2007) for video game
playing were shown to perform better with the addition of neuromodulation. In Norouzzadeh and Clune (2016), the authors
showed that neuromodulation produced forward models that
could adapt to changes significantly better than the controls.
They verified that evolution exploited variable learning rates
to perform adaptation when needed. In Velez and Clune
(2017), diffusion-based modulation, i.e., targeting entire parts
of the network, evolved to produce task-specific localized
learning and functional modularity, thus reducing the problem
of catastrophic forgetting.
The evidence in these studies suggests that neuromodulation
is a key ingredient to facilitate the evolution of learning in
EPANNs. They also indirectly suggest that neural systems
with more than one type of signal, e.g., activation and other
modulatory signals, are beneficial in the neuroevolution of
learning.

11

G. Indirectly encoded plasticity

A. Levels of abstraction and representations

An indirect genotype to phenotype mapping means that
evolution operates on a compact genotypical representation
(analogous to the DNA) that is then mapped into a fully
fledged network (analogous to a biological brain). Learning
rules may undergo a similar indirect mapping, so that compact
instructions in the genome expand to fully fledged plasticity
rules in the phenotype. One early study (Gruau and Whitley,
1993) encoded plasticity and development with a grammar
tree, and compared different learning rules on a simple
static task (parity and symmetry), demonstrating that learning
provided an evolutionary advantage in a static scenario. In
non-static contexts, and using a T-Maze domain as learning
task, Risi and Stanley (2010) showed that HyperNEAT, which
usually implements a compact encoding of weight patterns for
large-scale ANNs (Fig. 5a), can also encode patterns of local
learning rules. The approach, called adaptive HyperNEAT,
can encode arbitrary learning rules for each connection in an
evolving ANN based on a function of the ANN’s geometry
(Fig. 5b). Further flexibility was added in Risi and Stanley
(2012) to simultaneously encode the density and placement
of nodes in substrate space. The approach, called adaptive
evolvable-substrate HyperNEAT, makes it possible to indirectly encode plastic ANNs with thousands of connections
that exhibit regularities and repeating motifs. Adaptive ESHyperNEAT allows each individual synaptic connection, rather
than neuron, to be standard or modulatory, thus introducing
further design flexibility. Risi and Stanley (2014) showed how
adaptive HyperNEAT can be seeded to produce a specific
lateral connectivity pattern, thereby allowing the weights to
self-organize to form a topographic map of the input space.
The study shows that evolution can be seeded with specific
plasticity mechanisms that can facilitate the evolution of
specific types of learning.
The effect of indirectly encoded plasticity on the learning
and on the evolutionary process was investigated by Tonelli
and Mouret (2011, 2013). Using an operant conditioning task,
i.e., learning by reward, the authors showed that indirect
encodings that produced more regular neural structures also
improved the general EPANN learning abilities when compared to direct encodings. In an approach similar to adaptive
HyperNEAT, Orchard and Wang (2016) encoded the learning
rule itself as an evolving network. They named the approach
neural weights and bias update (NWB), and observed that
increasing the search space of the possible plasticity rules
created more general solutions than those based on only
Hebbian learning.

Choosing the right level of abstraction and the right representation (Bengio et al., 2013) are themes at the heart of many
problems in AI. In ANNs, low levels of abstraction are more
computationally expensive, but might be richer in dynamics.
High levels are faster to simulate, but require an intuition of
the essential dynamics that are necessary in the model.
One example that abstraction levels are still an open question derives from the comparison of two fields: deep learning
and bio-physical neural models. The performance of deep
networks (Krizhevsky et al., 2012; LeCun et al., 2015; Schmidhuber, 2015), based mainly on rate-based neurons and low
precision in the computation, appears to contrast with the lowlevel simulations of accurate bio-physical models. Examples
are the Blue Brain Project (Markram, 2006) and the Human
Brain Project (Human-Brain-Project, 2017). Although deep
learning and bio-physical models have different aims, they
both represent abstractions of neural computation. Research
in EPANNs is best placed to address the problem of levels
of abstraction because it can autonomously discover the most
appropriate level to evolve neural adaptation and intelligence.
Similarly to abstractions, representations play a critical
role. Compositional Patterns Producing Networks (CPPNs)
(Stanley, 2007), and also the previous work of Sims (1991),
demonstrated that structured phenotypes can be generated
through a function without going through the dynamic developmental process typical of multicellular organisms. Relatedly,
Hornby et al. (2002) showed that the different phenotypical
representations led to considerably different results in the
evolution of furniture. Miller (2014) discussed explicitly the
effect of abstraction levels for evolved developmental learning
networks, in particular in relation to two approaches that model
development at the neuron level or at the network level.
Finding appropriate abstractions and representations, just
as it was fundamental in the advances in deep learning to
represent input spaces and hierarchical features (Bengio et al.,
2013; Oquab et al., 2014), can also extend to representations of
internal models, learning mechanisms, and genetic encodings,
affecting the algorithms’ capabilities of evolving learning
abilities.

IV. F UTURE DIRECTIONS
The progress of EPANNs reviewed so far is based on
rapidly developing theories and technologies. In particular,
new advances in AI, machine learning, neural networks and
increased computational resources are currently creating a new
fertile research landscape, and are setting the groundwork for
new directions for EPANNs. This section presents promising
research themes that have the potential to extend and radically
change the field of EPANNs and AI as a whole.

B. Evolving general learning
One challenge in the evolution of learning is that evolved
learning may simply result in a switch among a finite set of
evolved behaviors, e.g., turning left or right in a T-Maze in a
finite sequence, which is all that evolving solutions encounter
during lifetime. A challenge for EPANNs is to acquire general
learning abilities in which the network is capable of learning problems not encountered during evolution. Mouret and
Tonelli (2014) propose the distinction between the evolution
of behavioral switches and the evolution of synaptic general
learning abilities, and suggest conditions that favor these types
of learning. General learning can be intuitively understood
as the capability to learn any association among input, internal, and output patterns, both in the spatial and temporal
dimensions, regardless of the complexity of the problem. Such
objective clearly poses practical and philosophical challenges.

12

(a) HyperNEAT

(b) Adaptive HyperNEAT

Figure 5: Example of an indirect mapping of plasticity rules from a compact genotype to a larger phenotype. (a) ANN nodes
in HyperNEAT are situated in physical space by assigning them specific coordinates. The connections between nodes are
determined by an evolved Compositional Patterns Producing Network (CPPN; Stanley (2007)), which takes as inputs the
coordinates of two ANN neurons and returns the weight between them. In the normal HyperNEAT approach (a), the CPPN is
queried once for all potential ANN connections when the agent is born. On the other hand, in adaptive HyperNEAT (b), the
CPPN is continually queried during the lifetime of the agent to determine individual connection weight changes based on the
location of neurons and additionally the activity of the presynaptic and postsynaptic neuron, and current connection weight.
Adaptive HyperNEAT is able to indirectly encode a pattern of nonlinear learning rules for each connection in the ANN (right).
Although humans are considered better at general learning
than machines, human learning skills are also specific and not
unlimited (Ormrod and Davis, 2004). Nevertheless, moving
from behavior switches to more general learning is a desirable
feature for EPANNs. Encouraging the emergence of general
learners may likely involve (1) an increased computational cost
for testing in rich environments, which include a large variety
of uncertain and stochastic scenarios with problems of various
complexity, and (2) an increased search space to explore the
evolution of complex strategies and avoid deception.
One promising direction is to evolve the ability to learn
algorithms and programs instead of simple associations. This
approach has recently shown promise in the context of deep
learning; methods such as the Neural Turing Machine (Graves
et al., 2014) can learn simple algorithms such as sort, copy,
etc. and approaches such as the Neural Programmer-Interface
(Reed and de Freitas, 2015) can learn to represent and execute
programs.
C. Incremental and social learning
An important open challenge for machine learning in general is the creation of neural systems that can continuously
integrate new knowledge and skills without forgetting what
they previously learned. A promising approach is progressive
neural networks (Rusu et al., 2016), in which a new network is created for each new task, and lateral connections
between networks allow the system to leverage previously
learned features. In the presence of time delays among stimuli,
actions and rewards, a rule called hypothesis testing plasticity
(HTP) (Soltoggio, 2015) implements fast and slow decay to
consolidate weights and suggests neural dynamics to avoid
catastrophic forgetting. A method to find the best shared
weights across multiple tasks, called elastic weight consolidation (EWC) was proposed in Kirkpatrick et al. (2017).
Plasticity rules that implement weight consolidation, given
their promise to prevent catastrophic forgetting, are likely to
become standard components in EPANNs.

Encouraging modularity (Ellefsen et al., 2015; Durr et al.,
2010) or augmenting evolving networks with a dedicated
external memory component (Lüders et al., 2016) have been
proposed recently. An evolutionary advantage is likely to
emerge for networks that can elaborate on previously learned
sub-skills during their lifetime to learn more complex tasks.
One interesting case in which incremental learning may play
a role is social learning (Best, 1999). EPANNs may learn both
from the environment and from other individuals, from scratch
or incrementally (Offerman and Sonnemans, 1998). In an
early study, McQuesten and Miikkulainen (1997) showed that
neuroevolution can benefit from parent networks teaching their
offspring through backpropagation. When social, learning may
involve imitation, language or communication, or other social
behaviors. Bullinaria (2017) proposes an EPANN framework
to simulate the evolution of culture and social learning. It
is reasonable to assume that future AI learning systems,
whether based on EPANNs or not, will acquire knowledge
through different modalities, involving direct experience with
the environment, but also social interaction, and possibly with
complex incremental learning phases.
D. Fast learning
Animal learning does not always require myriads of trials.
Humans can very quickly generalize from only a few given
examples, possibly leveraging previous experiences and a
long learning process during infancy. This type of learning,
advocated in AI and robotics systems (Thrun and Mitchell,
1995), is currently still missing in EPANNs. Inspiration for
new approaches could come from complementary learning
systems (McClelland et al., 1995; Kumaran et al., 2016) that
humans seem to possess, which include fast and slow learning
components. Additionally, approaches such as probabilistic
program induction seem to be able to learn concepts in oneshot at a human-level in some tasks (Lake et al., 2015). Fast
learning is likely to derive not just from trial-and-error, but also
from mental models that can be applied to diverse problems,

13

similarly to transfer learning (Thrun and Mitchell, 1995; Thrun
and O’Sullivan, 1996; Pan and Yang, 2010). Reusable mental
models, once learned, will allow agents to make predictions
and plan in new and uncertain scenarios with similarities
to previously learned ones. If EPANNs can discover neural
structures or learning rules that allow for generalization, an
evolutionary advantage of such a discovery will lead to its
full emergence and further optimization of such a property.
A rather different approach to accelerate learning was
proposed in Fernando et al. (2008); de Vladar and Szathmáry
(2015) and called Evolutionary Neurodynamics. According
to this theory, replication and selection might happen in a
neural system as it learns, thus mimicking an evolutionary
dynamics at the much faster time scale of a lifetime. We refer
to Fernando et al. (2012); de Vladar and Szathmáry (2015)
for an overview of the field. The appeal of this method is that
evolutionary search can be accelerated by implementing its
dynamics at both the evolution’s and life’s time-scales.
E. Evolving memory
The consequence of learning is memory, both explicit
and implicit (Anderson, 2013), and its consolidation (Dudai,
2012). For a review of computational models of memory see
Fusi (2017). EPANNs may reach solutions in which memory
evolved in different fashions, e.g., preserved as self-sustained
neural activity, encoded by connection weights modified by
plasticity rules, stored with an external memory (e.g. Neural
Turing Machine), or a combination of these approaches. Recurrent neural architectures based on long short-term memory
(LSTM) allow very complex tasks to be solved through
gradient descent training (Greff et al., 2015; Hochreiter and
Schmidhuber, 1997) and have recently shown promise when
combined with evolution (Rawal and Miikkulainen, 2016).
Neuromodulation and weight consolidation could also be used
to target areas of the network where information is stored.
Graves et al. (2014) introduced the Neural Turing Machine
(NTM), networks augmented with an external memory that
allows long-term memory storage. NTMs have shown promise
when trained through evolution (Greve et al., 2016; Lüders
et al., 2016, 2017) or gradient descent (Graves et al., 2014,
2016). The Evolvable Neural Turing Machine (ENTM) showed
good performance in solving the continuous version of the
double T-Maze navigation task (Greve et al., 2016), and
avoided catastrophic forgetting in a continual learning domain
(Lüders et al., 2016, 2017) because memory and control are
separated by design. Research in this area will reveal which
computational systems are more evolvable and how memories
will self organize and form in EPANNs.
F. EPANNs and deep learning
Deep learning has shown remarkable results in a variety
of different fields (Krizhevsky et al., 2012; Schmidhuber,
2015; LeCun et al., 2015). However, the model structures
of these networks are mostly hand-designed, include a large
number of parameters, and require extensive experiments to
discover optimal configurations. Advances in deep networks
have come from manually testing various design aspects such

as: different transfer functions, e.g., the ReLU (Krizhevsky
et al., 2012); generalization mechanisms, e.g. the dropout
(Srivastava et al., 2014); networks with different depth, e.g.
(Simonyan and Zisserman, 2014), and others (Schmidhuber,
2015). With increased computational resources, it is now
possible to search those design aspects with evolution.
Koutnı́k et al. (2014) used evolution to design a controller
that combined evolved recurrent neural networks, for the
control part, and a deep max-pooling convolutional neural
network to reduce the input dimensionality. The study does not
use evolution on the deep preprocessing networks itself, but
demonstrates nevertheless the evolutionary design of a deep
neural controller. Young et al. (2015) used an evolutionary
algorithm to optimize two parameters of a deep network: the
size (range [1,8]) and the number (range [16,126]) of the
filters in a convolutional neural network, showing that the optimized parameters could vary considerably from the standard
best-practice values. An established evolutionary computation
technique, the Covariance Matrix Adaptation Evolution Strategy (CMA-ES) Hansen and Ostermeier (2001), was used in
Loshchilov and Hutter (2016) to optimize the parameters of
a deep network to learn to classify the MNIST dataset. The
authors reported performance close to the state-of-the-art using
30 GPU devices.
Real et al. (2017) and Miikkulainen et al. (2017) showed
that evolutionary search can be used to determine the topology,
hyperparameters and building blocks of deep networks trained
through gradient descent. The performance were shown to rival
those of hand-designed architectures in the CIFAR-10 classification task and a language modeling task (Miikkulainen et al.,
2017), while Real et al. (2017) also tested the method on the
larger CIFAR-100 dataset. Desell (2017) proposes a method
called evolutionary exploration of augmenting convolutional
topologies, inspired by NEAT (Stanley and Miikkulainen,
2002), which evolves progressively more complex unstructured convolutional neural networks using genetically specified
feature maps and filters. This approach is also able to coevolve neural network training hyperparameters. Results were
obtained using 5 500 volunteered computers at the Citizen
Science Grid who were able to evolve competitive results
on the MNIST dataset in under 12 500 trained networks in
a period of approximately ten days. Liu et al. (2017) used
evolution to search for hierarchical architecture representations
showing competitive performance on the CIFAR-10 and Imagenet databases. The Evolutionary DEep Networks (EDEN)
framework (Dufourq and Bassett, 2017) aims to generalize
deep network optimization to a variety of problems and is
interfaced with TensorFlow (Abadi et al., 2016). A number of
similar software frameworks are currently being developed.
Fernando et al. (2017) used evolution to determine a subset of pathways through a network that are trained through
backpropagation, allowing the same network to learn a variety
of different tasks. Fernando et al. (2016) were also able to
rediscover convolutional networks by means of evolution of
Differentiable Pattern Producing Networks (Stanley, 2007).
In summary, evolutionary search allows not only to tune
hyperparameters of the networks, but also to combine different
learning rules in an innovative and counter-intuitive fashion.

14

Evolutionary algorithms are particularly suitable to operate
with unsupervised and reward-based problems. EPANNs need
not be differentiable, as in gradient descent based approaches,
thereby allowing a greater variety in architectures (Greve
et al., 2016). Finally, the deception in the evolution of learning suggests that gradient descent approaches can encounter
limitations when searching for complex learning strategies,
leaving EPANNs as a compelling alternative to discover more
advanced learning models.
G. GPU implementations and neuromorphic hardware
The progress of EPANNs will crucially depend on implementations that take advantage of the increased computational
power of parallel computation with GPUs and neuromorphic
hardware (Jo et al., 2010; Monroe, 2014). Deep learning
greatly benefited from GPU-accelerated machine learning but
also standardized tools (e.g. Torch, Tensorflow, Theano, etc.)
that made it easy for anybody to download, experiment, and
extend promising deep learning models.
Howard et al. (2011, 2012, 2014) devised experiments to
evolve plastic spiking networks implemented as memristors
for simulated robotic navigation tasks. Memristive plasticity
was observed consistently to enable higher performance than
constant-weighted connections in both static and dynamic
reward scenarios. Carlson et al. (2014) used GPU implementations to evolve plastic spiking neural networks with an
evolution strategy, which resulted in an efficient and automated
parameter tuning framework.
In the context of newly emerging technologies, it is worth
noting that, just as GPUs were not developed initially for
deep learning, so novel neural computation tools and hardware
systems, not developed for EPANNs, can now be exploited to
setup the articulated processes and computationally expensive
experiments required by EPANNs.
H. Measuring progress
The number of platforms and environments for testing
the capabilities of intelligent systems is constantly growing,
e.g., the Atari or General Video Game Playing Benchmark
(GVGAI, 2017), the Project Malmo (Microsoft, 2017), or the
OpenAI Universe, “a software platform for measuring and
training an AI’s general intelligence across the world’s supply
of games, websites and other applications” (OpenAI, 2017).
Measuring performance in supervised learning problems is
straight forward because the classification error can be used
as metrics. However, EPANNs are often evolved in rewardbased, survival, or novelty-oriented environments to discover
new, unknown, or creative learning strategies or behaviors.
Thus, desired behaviors or errors are not always defined.
Moreover, the goal for EPANNs is often not to be good at
solving one particular task, but rather to test the capability
to evolve the learning required for a range of problems, to
generalize to new problems, or to recover performance after
a change in the environment. Therefore, EPANNs will require
the community to devise and accept new metrics based on one
or more objectives such as the following:
• the time (in the evolutionary scale) to evolve the learning
mechanisms in one or more scenarios;

•
•
•
•
•
•

•
•
•

the time (in the lifetime scale) for learning in one or more
scenarios;
the number of different tasks that an EPANN evolves to
solve;
a measure of the variety of skills acquired by one
EPANN;
the complexity of the tasks and/or datasets, e.g., variations
in distributions, stochasticity, etc.;
the robustness and generalization capabilities of the
learner;
the recovery time in front of high-level variations or
changes, e.g., data distribution, type of problem, stochasticity levels, etc.;
computational resources used, e.g., number of lifetime
evaluations, length of a lifetime;
size, complexity, and computational requirements of the
solution once deployed.
novelty or richness of the behavior repertoire from multiple solutions, e.g., the variety of different EPANNs and
their strategies that were designed during evolution.

Few of those metrics are currently used to benchmark
machine learning algorithms. Research in EPANNs will foster
the adoption of such criteria as wider performance metrics for
assessing lifelong learning capabilities (Thrun and Pratt, 2012;
DARPA-L2M, 2017) of evolved plastic networks.
V. C ONCLUSION
This paper first described the broad inspiration and aspirations of evolved artificial plastic neural networks (EPANNs),
which are shown to draw from large, diverse, and interdisciplinary areas. The inspirations also reveal the ambitious
and long-term research objectives of EPANNs with important
implications for artificial intelligence and biology.
EPANNs saw considerable progress in the last two decades,
which led to the discovery of a number of further research
themes, such as the design of evolutionary algorithms to
promote the evolution of learning, a better understanding of the
complex interaction dynamics between evolution and learning,
the advantages of multi-signal networks such as modulatory
networks, and the attempt to evolve learning rules, architectures, and appropriate representations of learning mechanisms.
Finally, this paper outlined that recent scientific and technical progress may enable a step change in the potential of
EPANNs. In light of current advances, a pivotal moment for
EPANNs may be at hand as the next AI tool to find flexible algorithms, capable of discovering principles for general
adaptation and intelligent systems.
ACKNOWLEDGEMENTS
We thank John Bullinaria, Kris Carlson, Jeff Clune, Travis
Desell, Keith Downing, Dean Hougen, Joel Lehman, Jeff
Krichmar, Jay McClelland, Robert Merrison-Hort, Julian
Miller, Jean-Baptiste Mouret, James Stone, Eors Szathmary,
and Joanna Turner for insightful discussions and comments
on earlier versions of this paper.

15

R EFERENCES
Martı́n Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo,
Zhifeng Chen, Craig Citro, Greg S Corrado, Andy Davis,
Jeffrey Dean, Matthieu Devin, et al. Tensorflow: Large-scale
machine learning on heterogeneous distributed systems.
arXiv preprint arXiv:1603.04467, 2016.
L. F. Abbott. Modulation of Function and Gated Learning in a
Network Memory. Proceedings of the National Academy of
Science of the United States of America, 87(23):9241–9245,
1990.
Ajith Abraham. Meta learning evolutionary artificial neural
networks. Neurocomputing, 56:1–38, 2004.
Wickliffe C Abraham and Anthony Robins.
Memory
retention–the synaptic stability versus plasticity dilemma.
Trends in neurosciences, 28(2):73–78, 2005.
William H. Alexander and Olaf Sporns. An Embodied Model
of Learning, Plasticity, and Reward. Adaptive Behavior, 10
(3-4):143–159, 2002.
Louis Victor Allis et al. Searching for solutions in games and
artificial intelligence. Ponsen & Looijen, 1994.
Ethem Alpaydin. Introduction to machine learning. MIT press,
2014.
John R Anderson. The architecture of cognition. Psychology
Press, 2013.
Peter J Angeline, Gregory M Saunders, and Jordan B Pollack.
An evolutionary algorithm that constructs recurrent neural
networks. IEEE transactions on Neural Networks, 5(1):54–
65, 1994.
Jasmina Arifovic and Ramazan Gencay. Using genetic algorithms to select architecture of a feedforward artificial
neural network. Physica A: Statistical mechanics and its
applications, 289(3):574–594, 2001.
Solvi Arnold, Reiji Suzuki, and Takaya Arita. Evolution
of social representation in neural networks. In Advances
in Artificial Life, ECAL 2013: Proceedings of the Twelfth
European Conference on the Synthesis and Simulation of
Living Systems. MIT Press, Cambridge, MA, pages 425–
430, 2013a.
Solvi Arnold, Reiji Suzuki, and Takaya Arita. Selection
for reinforcement-free learning ability as an organizing
factor in the evolution of cognition. Advances in Artificial
Intelligence, 2013:8, 2013b.
Joshua Auerbach, Chrisantha Fernando, and Dario Floreano.
Online Extreme Evolutionary Learning Machines. In Proceedings of the Fourteenth International Conference on the
Synthesis and Simulation of Living Systems, 2014.
Nihat Ay, Jessica Flack, and David C Krakauer. Robustness
and complexity co-constructed in multimodal signalling networks. Philosophical Transactions of the Royal Society of
London B: Biological Sciences, 362(1479):441–447, 2007.
Thomas Bäck and Hans-Paul Schwefel. An overview of evolutionary algorithms for parameter optimization. Evolutionary
computation, 1(1):1–23, 1993.
Y. Bahroun, E. Hunsicker, and A. Soltoggio. Building Efficient
Deep Hebbian Networks for Image Classification Tasks.
In International Conference on Artificial Neural Networks,
2017.

C. H. Bailey, M. Giustetto, Y.-Y. Huang, R. D. Hawkins, and
E. R. Kandel. Is heterosynaptic modulation essential for
stabilizing Hebbian plasticity and memory? Nature Reviews
Neuroscience, 1(1):11–20, October 2000.
J Mark Baldwin. A new factor in evolution (continued).
American naturalist, pages 536–553, 1896.
Douglas A. Baxter, Carmen C. Canavier, John W. Clark, and
John H. Byrne. Computational Model of the Serotonergic
Modulation of Sensory Neurons in Aplysia. Journal of
Neurophysiology, 82:1914–2935, 1999.
Jonathan Baxter. The Evolution of Learning Algorithms for
Artificial Neural Networks. Complex Systems, D. Green
and T.Bossomaier, Eds. Amsterdam, The Netherlands: IOS,
pages 313–326, 1992.
Randall D. Beer and John C. Gallagher. Evolving dynamical
neural networks for adaptive behavior. Adaptive Behavior,
1(1):91–122, 1992.
S. Bengio, Y. Bengio, J. Cloutier, and J. Gecsei. On the
optimization of a synaptic learning rule. In Preprints Conf.
Optimality in Artificial and Biological Neural Networks,
Univ. of Texas, Dallas, Feb 6-8, 1992, 1992.
Yoshua Bengio, Samy Bengio, and Jocelyn Cloutier. Learning a synaptic learning rule. Technical report, Université
de Montréal, Département d’informatique et de recherche
opérationnelle, 1990.
Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learning: A review and new perspectives. IEEE
transactions on pattern analysis and machine intelligence,
35(8):1798–1828, 2013.
Peter Bentley. Evolutionary design by computers. Morgan
Kaufmann, 1999.
Michael L Best. How culture can guide evolution: An inquiry
into gene/meme enhancement and opposition. Adaptive
Behavior, 7(3-4):289–306, 1999.
L. Elie Bienenstock, Leon N. Cooper, and Paul W. Munro.
Theory for the development of neuron selectivity: orientation specificity and binocular interaction in visual cortex.
The Journal of Neuroscience, 2(1):32–48, January 1982.
John T. Birmingham. Increasing Sensor Flexibility Through
Neuromodulation. Biological Bulletin, 200:206–210, April
2001.
Jesper Blynel and Dario Floreano. Levels of Dynamics and
Adaptive Behavior in Evolutionary Neural Controllers. In
Proceedings of the seventh international conference on simulation of adaptive behavior on From animals to animats,
pages 272–281. MIT Press Cambridge, MA, USA, 2002.
Jesper Blynel and Dario Floreano. Exploring the T-Maze:
Evolving Learning-Like Robot Behaviors Using CTRNNs.
In EvoWorkshops, pages 593–604, 2003.
Egbert JW Boers, Marko V Borst, and Ida G SprinkhuizenKuyper. Evolving neural networks using the “Baldwin
effect”. In Artificial Neural Nets and Genetic Algorithms,
pages 333–336. Springer, 1995.
Hervé Bourlard and Yves Kamp. Auto-association by multilayer perceptrons and singular value decomposition. Biological cybernetics, 59(4-5):291–294, 1988.

16

Thomas H. Brown, Edward W. Kairiss, and Claude L.
Keenan. Hebbian Synapse: Biophysical Mechanisms and
Algorithms. Annual Review of Neuroscience, 13:475–511,
1990.
John A Bullinaria. Exploring the Baldwin effect in evolving
adaptable control systems. In Connectionist models of learning, development and evolution, pages 231–242. Springer,
2001.
John A Bullinaria. From biological models to the evolution of
robot control systems. Philosophical Transactions: Mathematical, Physical and Engineering Sciences, pages 2145–
2164, 2003.
John A Bullinaria. The effect of learning on life history
evolution. In Proceedings of the 9th annual conference
on Genetic and evolutionary computation, pages 222–229.
ACM, 2007a.
John A Bullinaria. Understanding the emergence of modularity
in neural systems. Cognitive science, 31(4):673–695, 2007b.
John A Bullinaria. Evolved Dual Weight Neural Architectures
to Facilitate Incremental Learning. In IJCCI, pages 427–
434, 2009a.
John A Bullinaria. The importance of neurophysiological constraints for modelling the emergence of modularity. Computational modelling in behavioural neuroscience: closing the
gap between neurophysiology and behaviour, 2:188–208,
2009b.
John A Bullinaria. Lifetime learning as a factor in life history
evolution. Artificial Life, 15(4):389–409, 2009c.
John A. Bullinaria. Imitative and Direct Learning as Interacting Factors in Life History Evolution. Artificial Life, 23,
2017.
Jeremie Cabessa and Hava T Siegelmann. The super-Turing
computational power of plastic recurrent neural networks.
International journal of neural systems, 24(08):1450029,
2014.
Thomas J. Carew, Edgar T. Walters, and Eric R. Kandel.
Classical conditioning in a simple withdrawal reflex in
Aplysia californica. The Journal of Neuroscience, 1(12):
1426–1437, December 1981.
Kristofor D Carlson, Jayram Moorkanikara Nageswaran, Nikil
Dutt, and Jeffrey L Krichmar. An efficient automated
parameter tuning framework for spiking neural networks.
Neuromorphic Engineering Systems and Applications, 8,
2014.
Charles S Carver and Michael F Scheier. Attention and selfregulation: A control-theory approach to human behavior.
Springer Science & Business Media, 2012.
D Cervier. AI: The Tumultuous Search for Artificial Intelligence, 1993.
David J Chalmers. The evolution of learning: An experiment
in genetic connectionism. In Proceedings of the 1990
connectionist models summer school, pages 81–90. San
Mateo, CA, 1990.
Dmitri B Chklovskii, BW Mel, and K Svoboda. Cortical
rewiring and information storage. Nature, 431(7010):782–
788, 2004.

Gregory A. Clark and Eric R. Kandel. Branch-specific
heterosynaptic facilitation in Aplysia siphon sensory cells.
PNAS, 81(8):2577–2581, 1984.
Dave Cliff, Phil Husbands, and Inman Harvey. Explorations
in Evolutionary Robotics. Adaptive Behavior, 2(1):73–110,
1993.
Tim H Clutton-Brock. The evolution of parental care. Princeton University Press, 1991.
Oliver J Coleman and Alan D Blair. Evolving plastic neural
networks for online learning: review and future directions.
In AI 2012: Advances in Artificial Intelligence, pages 326–
337. Springer, 2012.
Steven J. Cooper. Donald O. Hebb’s synapse and learning rule:
a history and commentary. Neuroscience and Biobehavioral
Reviews, 28(8):851–874, January 2005.
Antonio R Damasio. The feeling of what happens: Body and
emotion in the making of consciousness. Houghton Mifflin
Harcourt, 1999.
DARPA-L2M.
DARPA, Lifelong Learning Machines.
https://www.fbo.gov/spg/ODA/DARPA/CMO/
HR001117S0016/listing.html, accessed July 2017, 2017.
Charles Darwin. On the origin of species by means of natural
selection, or The preservation of favoured races in the
struggle for life. Murray, London, 1859.
Richard Dawkins. The evolution of evolvability. On growth,
form and computers, pages 239–255, 2003.
Alain De Botton. Six areas that artificial emotional intelligence
will revolutionise. http://www.wired.co.uk/article/wiredworld-2016-alain-de-botton-artificial-emotionalintelligence, 2016. Accessed: 12/11/2016.
Harold P de Vladar and Eörs Szathmáry. Neuronal boost
to evolutionary dynamics. Interface focus, 5(6):20150074,
2015.
Ian J Deary, Wendy Johnson, and Lorna M Houlihan. Genetic
foundations of human intelligence. Human genetics, 126
(1):215–232, 2009.
Li Deng, Geoffrey Hinton, and Brian Kingsbury. New types
of deep neural network learning for speech recognition and
related applications: An overview. In Acoustics, Speech
and Signal Processing (ICASSP), 2013 IEEE International
Conference on, pages 8599–8603. IEEE, 2013.
Travis Desell. Large Scale Evolution of Convolutional Neural
Networks Using Volunteer Computing. arXiv preprint
arXiv:1703.05422, 2017.
Ezequiel Di Paolo. Spike-timing dependent plasticity for
evolved robots. Adaptive Behavior, 10(3-4):243–263, 2002.
Ezequiel A. Di Paolo. Evolving spike-timing-depending
plasticity for single-trial learning in robots. Philosophical
Transactions of the Royal Society, 361(1811):2299–2319,
October 2003.
Theodosius Dobzhansky. Genetics of the evolutionary process,
volume 139. Columbia University Press, 1970.
Norman Doidge. The brain that changes itself: Stories of personal triumph from the frontiers of brain science. Penguin,
2007.
Keith L. Downing. Supplementing Evolutionary Development Systems with Abstract Models of Neurogenesis. In

17

Dirk Thiersens at al., editor, Proceeding of the Genetic
and Evolutionary Computation Conference (GECCO 2007),
pages 990–998. ACM, 2007.
Keith L Downing. Intelligence Emerging: Adaptivity and
Search in Evolving Neural Systems. MIT Press, 2015.
Kenji Doya. Metalearning and neuromodulation. Neural
Networks, 15(4-6):495–506, 2002.
Kenji Doya and Eiji Uchibe. The Cyber Rodent Project:
Exploration and Adaptive Mechanisms for Self-Preservation
and Self-Reproduction. Adaptive Behavior, 13(2):149–160,
2005.
Bogdam Draganski and Arne May. Training-induced structural
changes in the adult human brain. Behavioural brain
research, 192(1):137–142, 2008.
Yadin Dudai. The restless engram: consolidations never end.
Annual review of neuroscience, 35:227–247, 2012.
Emmanuel Dufourq and Bruce A Bassett. EDEN: Evolutionary Deep Networks for Efficient Machine Learning. arXiv
preprint arXiv:1709.09161, 2017.
Peter Durr, Dario Floreano, and Claudio Mattiussi. Genetic representation and evolvability of modular neural
controllers. IEEE Computational Intelligence Magazine, 5
(3):10–19, 2010.
Gerald M Edelman and Giulio Tononi. A universe of consciousness: How matter becomes imagination. Basic books,
2000.
Agoston E Eiben and Jim Smith. From evolutionary computation to the evolution of things. Nature, 521(7553):476–482,
2015.
Kai Olav Ellefsen. Different regulation mechanisms for
evolved critical periods in learning. In 2013 IEEE Congress
on Evolutionary Computation, pages 1193–1200. IEEE,
2013a.
Kai Olav Ellefsen. Evolved sensitive periods in learning.
Advances in Artificial Life, ECAL 2013: Proceedings of
the Twelfth European Confer- ence on the Synthesis and
Simulation of Living Systems, 12:409–416, 2013b.
Kai Olav Ellefsen. The Evolution of Learning Under Environmental Variability. Artificial Life 14: Proceedings of The
Fourteenth International Conference on the Synthesis and
Simulation of Living Systems, 2014.
Kai Olav Ellefsen, Jean-Baptiste Mouret, and Jeff Clune.
Neural Modularity Helps Organisms Evolve to Learn New
Skills without Forgetting Old Skills. PLoS Comput Biol, 11
(4):e1004128, 2015.
Brent E Eskridge and Dean F Hougen. Nurturing promotes
the evolution of learning in uncertain environments. In Development and Learning and Epigenetic Robotics (ICDL),
2012 IEEE International Conference on, pages 1–6. IEEE,
2012.
Diego Federici. Evolving Developing Spiking Neural Networks. In Proceedings of the IEEE Congress on Evolutionary Computation, CEC 2005, 2005.
Jean-Marc Fellous and Christiane Linster. Computational
Models of Neuromodulation. Neural Computation, 10:771–
805, 1998.

Chrisantha Fernando, KK Karishma, and Eörs Szathmáry.
Copying and evolution of neuronal topology. PloS one, 3
(11):e3775, 2008.
Chrisantha Fernando, Eörs Szathmáry, and Phil Husbands.
Selectionist and evolutionary approaches to brain function: a
critical appraisal. Frontiers in computational neuroscience,
6, 2012.
Chrisantha Fernando, Dylan Banarse, Malcolm Reynolds,
Frederic Besse, David Pfau, Max Jaderberg, Marc Lanctot, and Daan Wierstra. Convolution by Evolution: Differentiable Pattern Producing Networks. arXiv preprint
arXiv:1606.02580, 2016.
Chrisantha Fernando, Dylan Banarse, Charles Blundell, Yori
Zwols, David Ha, Andrei A Rusu, Alexander Pritzel,
and Daan Wierstra. PathNet: Evolution Channels Gradient Descent in Super Neural Networks. arXiv preprint
arXiv:1701.08734, 2017.
Peter SB Finnie and Karim Nader. The role of metaplasticity
mechanisms in regulating memory destabilization and reconsolidation. Neuroscience & Biobehavioral Reviews, 36
(7):1667–1707, 2012.
D. Floreano and F. Mondada. Evolution of Homing Navigation
in a Real Mobile Robot. IEEE Transactions on Systems,
Man, and Cybernetics–Part B, 26(3):396–407, 1996.
D. Floreano and J. Urzelai. Neural morphogenesis, synaptic
plasticity, and evolution. Theory in Biosciences, 240:225–
240, 2001a.
Dario Floreano and Claudio Mattiussi. Bio-inspired artificial
intelligence: theories, methods, and technologies. MIT
press, 2008.
Dario Floreano and Francesco Mondada. Automatic creation
of an autonomous agent: genetic evolution of a neuralnetwork driven robot. In Proceedings of the third international conference on Simulation of adaptive behavior :
from animals to animats 3, pages 421–430. MIT Press,
Cambridge, MA, USA, 1994.
Dario Floreano and Stefano Nolfi. Evolutionary Robotics. The
MIT Press, 2004.
Dario Floreano and Joseba Urzelai. Evolution of Plastic
Control Networks. Auton. Robots, 11(3):311–317, 2001b.
David B Fogel. Evolutionary computation: toward a new
philosophy of machine intelligence, volume 1. John Wiley
& Sons, 2006.
JF Fontanari and R Meir. Evolving a learning algorithm for
the binary perceptron. Network: Computation in Neural
Systems, 2(4):353–359, 1991.
Akinobu Fujii, Nobuhiro Saito, Kota Nakahira, and Akio
Ishiguro. Generation of an Adaptive Controller CPG for
a Quadruped Robot with Neuromodulation Mechanism. In
Proceedings of the 2002 IEEE/RSJ International Conference On Intelligent Robots and Systems, EPFL, Lausanne,
Switzerland, pages 2619–2624, 2002.
Stefano Fusi. Computational models of long term plasticity
and memory. arXiv preprint arXiv:1706.04946, 2017.
Wulfram Gerstner and Werner M Kistler. Mathematical
formulations of Hebbian learning. Biological cybernetics,
87(5-6):404–415, 2002a.

18

Wulfram Gerstner and Werner M Kistler. Spiking neuron
models: Single neurons, populations, plasticity. Cambridge
university press, 2002b.
David E Goldberg and John H Holland. Genetic algorithms
and machine learning. Machine learning, 3(2):95–99, 1988.
Alex Graves, Greg Wayne, and Ivo Danihelka. Neural Turing
machines. arXiv preprint arXiv:1410.5401, 2014.
Alex Graves, Greg Wayne, Malcolm Reynolds, Tim
Harley, Ivo Danihelka, Agnieszka Grabska-Barwińska, Sergio Gómez Colmenarejo, Edward Grefenstette, Tiago Ramalho, John Agapiou, et al. Hybrid computing using a
neural network with dynamic external memory. Nature, 538
(7626):471–476, 2016.
Klaus Greff, Rupesh Kumar Srivastava, Jan Koutnı́k, Bas R
Steunebrink, and Jürgen Schmidhuber. LSTM: A search
space odyssey. arXiv preprint arXiv:1503.04069, 2015.
Rasmus Boll Greve, Emil Juul Jacobsen, and Sebastian Risi.
Evolving Neural Turing Machines for Reward-based Learning. In Proceedings of the Genetic and Evolutionary
Computation Conference 2016, GECCO ’16, pages 117–
124, New York, NY, USA, 2016. ACM. ISBN 978-1-45034206-3.
ST Grossberg. Studies of mind and brain: Neural principles
of learning, perception, development, cognition, and motor
control, volume 70. Springer Science & Business Media,
2012.
Frederic Gruau and Darrell Whitley. Adding learning to the
cellular development of neural networks: Evolution and the
Baldwin effect. Evolutionary computation, 1(3):213–233,
1993.
Q. Gu. Neuromodulatory transmitter systems in the cortex
and their role in cortical plasticity. Neuroscience, 111(4):
815–853, 2002.
Jimmy Gustafsson. Evolving neuromodulatory topologies for
plasticity in video game playing. Master’s thesis, Blekinge
Institute of Technology, Faculty of Computing, Department
of Software Engineering, 2016.
GVGAI.
The General Video game AI Competition.
http://www.gvgai.net, 2017. Accessed: 21/01/2017.
Nikolaus Hansen and Andreas Ostermeier. Completely derandomized self-adaptation in evolution strategies. Evolutionary computation, 9(2):159–195, 2001.
Bart LM Happel and Jacob MJ Murre. Design and evolution
of modular neural network architectures. Neural networks,
7(6):985–1004, 1994.
Kyle Ira Harrington, Emmanuel Awa, Sylvain Cussat-Blanc,
and Jordan Pollack. Robot coverage control by evolved
neuromodulation. In Neural Networks (IJCNN), The 2013
International Joint Conference on, pages 1–8. IEEE, 2013.
Ronald M. Harris-Warrick and Eve Marder. Modulation of
neural networks for behavior. Annual Review of Neuroscience, 14:39–57, 1991.
Inman Harvey, Phil Husbands, Dave Cliff, Adrian Thompson,
and Nick Jakobi. Evolutionary robotics: the Sussex approach. Robotics and autonomous systems, 20(2):205–224,
1997.

Michael E. Hasselmo. Neuromodulation and cortical function:
modeling the physiological basis of behavior. Behavioural
Brain Research, 67:1–27, 1995.
Michael E. Hasselmo. Neuromodulation: acetylcholine and
memory consolidation. Trends in Cognitive Sciences, 3:
351–359, September 1999.
Michael E. Hasselmo and Eric Schnell. Laminar selectivity
of the cholinergic suppression of synaptic transmission in
rat hippocampal region CA1: computational modeling and
brain slice physiology. Journal of Neuroscience, 14(6):
3898–3914, 1994.
Jeff Hawkins and Sandra Blakeslee. On intelligence. Macmillan, 2007.
Donald Olding Hebb. The Organization of Behavior: A
Neuropsychological Theory. Wiley, New York, 1949.
Takao K Hensch, Michela Fagiolini, Nobuko Mataga,
Michael P Stryker, Steinunn Baekkeskov, and Shera F
Kash. Local GABA circuit control of experience-dependent
plasticity in developing visual cortex. Science, 282(5393):
1504–1508, 1998.
Geoffrey E. Hinton and Steven J. Nowlan. How Learning Can
Guide Evolution. Complex Systems, 1:495–502, 1987.
Geoffrey E Hinton and David C Plaut. Using fast weights to
deblur old memories. In Proceedings of the ninth annual
conference of the Cognitive Science Society, pages 177–186,
1987.
Geoffrey E Hinton and Ruslan R Salakhutdinov. Reducing the
dimensionality of data with neural networks. Science, 313
(5786):504–507, 2006.
Sepp Hochreiter and Jürgen Schmidhuber. Long Short-Term
Memory. Neural computation, 9(8):1735–1780, 1997.
Thierry Hoinville, Cecilia Tapia Siles, and Patrick Hénaff.
Flexible and multistable pattern generation by evolving
constrained plastic neurocontrollers. Adaptive Behavior, 19
(3):187–207, 2011.
John H Holland. Adaptation in natural and artificial systems:
an introductory analysis with applications to biology, control, and artificial intelligence. U Michigan Press, 1975.
John H Holland and Judith S Reitman. Cognitive systems
based on adaptive algorithms. ACM SIGART Bulletin, 63:
49–49, 1977.
William D Hopkins, Jamie L Russell, and Jennifer Schaeffer.
Chimpanzee intelligence is heritable. Current Biology, 24
(14):1649–1652, 2014.
Gregory Hornby, Jordan Pollack, et al. Creating high-level
components with a generative representation for body-brain
evolution. Artificial life, 8(3):223–246, 2002.
Gerard Howard, Ella Gale, Larry Bull, Ben de Lacy Costello,
and Andrew Adamatzky. Towards evolving spiking networks with memristive synapses. In Artificial Life (ALIFE),
2011 IEEE Symposium on, pages 14–21. IEEE, 2011.
Gerard Howard, Ella Gale, Larry Bull, Ben de Lacy Costello,
and Andy Adamatzky. Evolution of plastic learning in
spiking networks via memristive connections. IEEE Transactions on Evolutionary Computation, 16(5):711–729, 2012.
Gerard Howard, Larry Bull, Ben de Lacy Costello, Ella Gale,
and Andrew Adamatzky. Evolving spiking networks with

19

variable resistive memories. Evolutionary computation, 22
(1):79–103, 2014.
C. L. Hull. Principles of behavior. New-Your: Appleton
Century, 1943.
Human-Brain-Project. Human Brain Project. https://www.
humanbrainproject.eu/en/, accessed July 2017, 2017.
Phil Husbands, Tom Smith, Nick Jakobi, and Michael O’Shea.
Better Living Through Chemistry: Evolving GasNet for
Robots Control. Connection Science, 10:185–210, 1998.
Eugene M Izhikevich. Polychronization: computation with
spikes. Neural computation, 18(2):245–282, 2006.
Eugene M Izhikevich. Solving the distal reward problem
through linkage of STDP and dopamine signaling. Cerebral
cortex, 17(10):2443–2452, 2007.
Herbert Jaeger and Harald Haas. Harnessing nonlinearity:
Predicting chaotic systems and saving energy in wireless
communication. science, 304(5667):78–80, 2004.
Sung Hyun Jo, Ting Chang, Idongesit Ebong, Bhavitavya B
Bhadviya, Pinaki Mazumder, and Wei Lu. Nanoscale memristor device as synapse in neuromorphic systems. Nano
letters, 10(4):1297–1301, 2010.
E. R. Kandel and L. Tauc. Heterosynaptic facilitation in
neurones of the abdominal ganglion of Aplysia depilans.
The Journal of Physiology, 181:1–27, 1965.
Eric R. Kandel. In Search of Memory - The Emergence of a
New Science of Mind. WW Norton & Company, New York,
2007.
Paul S. Katz. Intrinsic and extrinsic neuromodulation of
motor circuits. Current Opinion in Neurobiology, 5:799–
808, 1995.
Paul S. Katz and William N. Frost. Intrinsic neuromodulation:
altering neuronal circuits from within. Trends in Neurosciences, 19:54–61, 1996.
Paul S. Katz, Peter A. Getting, and William N. Frost. Dynamic
neuromodulation of synaptic strength intrinsic to a central
pattern generator circuit. Nature, 367:729–731, 1994.
Gul Muhammad Khan and Julian Francis Miller. In search of
intelligence: evolving a developmental neuron capable of
learning. Connection Science, 26(4):297–333, 2014.
Gul Muhammad Khan, Julian F. Miller, and David M. Halliday. Coevolution of neuro-developmental programs that
play checkers. In Springer, editor, Proceedings of 8th
International Conference on Evolvable Systems (ICES):
From Biology to Hardware, volume 5216, pages 352–361,
2008.
Gul Muhammad Khan, Julian F. Miller, and David M. Halliday. Evolution of Cartesian Genetic Programs for Development of Learning Neural Architecture. Evolutionary
Computation, 19(3):469–523, 2011a.
Maryam Mahsal Khan, Gul Muhammad Khan, and Julian F.
Miller. Developmental Plasticity in Cartesian Genetic
Programming based Neural Networks. In ICINCO 2011
- Proceedings of the 8th International Conference on Informatics in Control, Automation and Robotics, Volume
1, Noordwijkerhout, The Netherlands, 28 - 31 July, 2011,
pages 449–458, 2011b.

James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A. Rusu, Kieran Milan,
John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska,
Demis Hassabis, Claudia Clopath, Dharshan Kumaran, and
Raia Hadsell. Overcoming catastrophic forgetting in neural
networks. Proceedings of the National Academy of Sciences,
2017.
Tomomi Kiyota. Neurogenesis and Brain Repair. In Neuroimmune Pharmacology, pages 575–597. Springer, 2017.
Hope Klug and Michael B Bonsall. Life history and the
evolution of parental care. Evolution, 64(3):823–835, 2010.
Teuvo Kohonen. Self-organized formation of topologically
correct feature maps. Biological cybernetics, 43(1):59–69,
1982.
Teuvo Kohonen. The Self-Organizing Map. Proceedings of
the IEEE, 78(9):1464–1480, 1990.
Bryan Kolb. Brain development, plasticity, and behavior.
American Psychologist, 44(9):1203, 1989.
Bryan Kolb and Robbin Gibb. Brain plasticity and behaviour
in the developing brain. Journal of the Canadian Academy
of Child & Adolescent Psychiatry, 20(4), 2011.
Toshiyuki Kondo. Evolutionary design and behaviour analysis
of neuromodulatory neural networks for mobile robots control. Applied Soft Computing, 7(1):189–202, January 2007.
Jan Koutnı́k, Jürgen Schmidhuber, and Faustino Gomez.
Evolving deep unsupervised convolutional networks for
vision-based reinforcement learning. In Proceedings of
the 2014 Annual Conference on Genetic and Evolutionary
Computation, pages 541–548. ACM, 2014.
John R Koza. Genetic programming: on the programming of
computers by means of natural selection, volume 1. MIT
press, 1992.
Jeffrey L Krichmar. The neuromodulatory system: a framework for survival and adaptive behavior in a challenging
world. Adaptive Behavior, 16(6):385–399, 2008.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton.
Imagenet classification with deep convolutional neural networks. In Advances in neural information processing
systems, pages 1097–1105, 2012.
Dharshan Kumaran, Demis Hassabis, and James L McClelland. What learning systems do intelligent agents need?
complementary learning systems theory updated. Trends in
Cognitive Sciences, 20(7):512–534, 2016.
Irving Kupfermann. Cellular Neurobiology: Neuromodulation.
Science, 236:863, 1987.
SIMO KϕPPE. The psychology of the neuron: Freud, Cajal
and Golgi. Scandinavian journal of psychology, 24(1):1–12,
1983.
Brenden M Lake, Ruslan Salakhutdinov, and Joshua B Tenenbaum. Human-level concept learning through probabilistic
program induction. Science, 350(6266):1332–1338, 2015.
Alexander Lalejini and Charles Ofria. The Evolutionary
Origins of Phenotypic Plasticity. In Proceedings of the
Artificial Life Conference, 2016.
Raphael Lamprecht and Joseph LeDoux. Structural plasticity
and memory. Nature Reviews Neuroscience, 5(1):45–54,
2004.

20

Christopher G Langton. Artificial life: An overview. Mit Press,
1997.
Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep
learning. Nature, 521(7553):436–444, 2015.
Joseph E LeDoux. Synaptic self: How our brains become who
we are. Penguin, 2003.
Chul Min Lee and Shrikanth S Narayanan. Toward detecting
emotions in spoken dialogs. IEEE transactions on speech
and audio processing, 13(2):293–303, 2005.
Joel Lehman and Risto Miikkulainen. Overcoming Deception in Evolution of Cognitive Behaviors. In Proceedings
of the Genetic and Evolutionary Computation Conference
(GECCO 2014), Vancouver, BC, Canada, July 2014.
Joel Lehman and Kenneth O Stanley. Exploiting OpenEndedness to Solve Problems Through the Search for Novelty. In ALIFE, pages 329–336, 2008.
Joel Lehman and Kenneth O Stanley. Abandoning objectives:
Evolution through the search for novelty alone. Evolutionary computation, 19(2):189–223, 2011.
Joel Lehman and Kenneth O Stanley. Evolvability is inevitable: increasing evolvability without the pressure to
adapt. PloS one, 8(4):e62186, 2013.
Joseph P Levy and Dimitrios Bairaktaris. Connectionist dualweight architectures. Language and Cognitive Processes,
10(3-4):265–283, 1995.
Long-Ji Lin. Reinforcement Learning for Robots Using Neural Networks. PhD thesis, School of Computer Science,
Carnegie Mellon University, 1993.
Hanxiao Liu, Karen Simonyan, Oriol Vinyals, Chrisantha
Fernando, and Koray Kavukcuoglu. Hierarchical representations for efficient architecture search. arXiv preprint
arXiv:1711.00436, 2017.
Ilya Loshchilov and Frank Hutter. CMA-ES for Hyperparameter Optimization of Deep Neural Networks. arXiv preprint
arXiv:1604.07269, 2016.
Benno Lüders, Mikkel Schläger, and Sebastian Risi. Continual
Learning through Evolvable Neural Turing Machines. In
Proceedings of the NIPS 2016 Workshop on Continual
Learning and Deep Networks (CLDL 2016), 2016.
Benno Lüders, Mikkel Schläger, Aleksandra Korach, and
Sebastian Risi. Continual and One-Shot Learning through
Neural Networks with Dynamic External Memory. In
Proceedings of the 20th European Conference on the Applications of Evolutionary Computation (EvoApplications
2017), 2017.
Wolfgang Maass and Christopher M Bishop. Pulsed neural
networks. MIT press, 2001.
Michail Maniadakis and Panos Trahanias. Modelling brain
emergent behaviours through coevolution of neural agents.
Neural Networks, 19(5):705–720, 2006.
Eve Marder. Neural modulation: Following your own rhythm.
Current Biology, 6(2):119–121, 1996.
Eve Marder and Vatsala Thirumalai. Cellular, synaptic and
network effects of neuromodulation. Neural Networks, 15:
479–493, 2002.
Henry Markram. The Blue Brain Project. Nature Reviews
Neuroscience, 7(2):153–160, 2006.

Henry Markram, Joachim Lübke, Michael Frotscher, and Bert
Sakmann. Regulation of synaptic efficacy by coincidence of
postsynaptic APs and EPSPs. Science, 275(5297):213–215,
1997.
Claudio Mattiussi and Dario Floreano. Analog genetic encoding for the evolution of circuits and networks. IEEE
Transactions on evolutionary computation, 11(5):596–607,
2007.
Giles Mayley. Landscapes, learning costs, and genetic assimilation. Evolutionary Computation, 4(3):213–234, 1996.
Giles Mayley. Guiding or hiding: Explorations into the effects
of learning on the rate of evolution. In Proceedings of the
Fourth European Conference on Artificial Life, volume 97,
pages 135–144, 1997.
James L McClelland, Bruce L McNaughton, and Randall C
O’reilly. Why there are complementary learning systems
in the hippocampus and neocortex: insights from the successes and failures of connectionist models of learning and
memory. Psychological review, 102(3):419, 1995.
Paul McQuesten and Risto Miikkulainen. Culling and Teaching in Neuro-Evolution. In ICGA, pages 760–767, 1997.
Yan Meng, Yaochu Jin, and Jun Yin. Modeling activitydependent plasticity in BCM spiking neural networks with
application to human behavior recognition. IEEE transactions on neural networks, 22(12):1952–1966, 2011.
Frederic Mery and James G Burns. Behavioural plasticity: an
interaction between evolution and experience. Evolutionary
Ecology, 24(3):571–583, 2010.
Michael M Merzenich, Randall J Nelson, Michael P Stryker,
Max S Cynader, Axel Schoppmann, and John M Zook.
Somatosensory cortical map changes following digit amputation in adult monkeys. Journal of comparative neurology,
224(4):591–605, 1984.
Zbigniew Michalewicz. GAs: What are they? In Genetic
algorithms+ data structures= evolution programs, pages
13–30. Springer, 1994.
Ryszard S Michalski, Jaime G Carbonell, and Tom M
Mitchell. Machine learning: An artificial intelligence approach. Springer Science & Business Media, 2013.
Thomas Miconi. The road to everywhere: Evolution, complexity and progress in natural and artificial systems. PhD
thesis, University of Birmingham, 2008.
Microsoft. Project malmo. https://github.com/Microsoft/
malmo, accessed Nov 2017, 2017.
Risto Miikkulainen, Jason Liang, Elliot Meyerson, Aditya
Rawal, Dan Fink, Olivier Francon, Bala Raju, Arshak
Navruzyan, Nigel Duffy, and Babak Hodjat. Evolving Deep
Neural Networks. arXiv preprint arXiv:1703.00548, 2017.
Julian F Miller. Neuro-Centric and Holocentric Approaches to
the Evolution of Developmental Neural Networks. In Growing Adaptive Machines, pages 227–249. Springer, 2014.
Kenneth D. Miller and David J. C. Mackay. The Role of
Constraints in Hebbian Learning. Neural Computation, 6:
100–126, 1994.
Ian Millington and John Funge. Artificial intelligence for
games. CRC Press, 2016.

21

Don Monroe. Neuromorphic computing gets ready for the
(really) big time. Communications of the ACM, 57(6):13–
15, 2014.
David S Moore. The Dependent Gene: The Fallacy of “Nature
vs. Nurture”. 2003.
Jean-Baptiste Mouret and Paul Tonelli. Artificial evolution of
plastic neural networks: a few key concepts. In Growing
Adaptive Machines, pages 251–261. Springer, 2014.
Yael Niv, Daphna Joel, Isaac Meilijson, and Eytan Ruppin.
Evolution of Reinforcement Learning in Uncertain Environments: A Simple Explanation for Complex Foraging
Behaviours. Adaptive Behavior, 10(1):5–24, 2002.
B Nogueira, Y Lenon, C Eduardo, Creto A Vidal, and
Joaquim B Cavalcante Neto. Evolving plastic neuromodulated networks for behavior emergence of autonomous
virtual characters. In Advances in Artificial Life, ECAL
2013: Proceedings of the Twelfth European Conference on
the Synthesis and Simulation of Living Systems. MIT Press,
Cambridge, MA, pages 577–584, 2013.
Stefano Nolfi and Dario Floreano. Learning and evolution.
Autonomous robots, 7(1):89–113, 1999.
Stefano Nolfi and Dario Floreano. Evolutionary robotics,
2000.
Stefano Nolfi and Domenico Parisi. Auto-teaching: networks
that develop their own teaching input. In Free University of
Brussels. Citeseer, 1993.
Stefano Nolfi and Domenico Parisi. Learning to adapt to
changing environments in evolving neural networks. Adaptive behavior, 5(1):75–98, 1996.
Mohammad Sadegh Norouzzadeh and Jeff Clune. Neuromodulation Improves the Evolution of Forward Models. In
Proceedings of the Genetic and Evolutionary Computation
Conference, 2016.
Theo Offerman and Joep Sonnemans. Learning by experience
and learning by imitating successful others. Journal of
economic behavior & organization, 34(4):559–575, 1998.
Erkki Oja. A Simplified Neuron Model as a Principal Component Analyzer. Journal of Mathematical Biology, 15(3):
267–273, November 1982.
OpenAI. OpenAI Universe. https://openai.com/blog/universe/,
2017. Accessed: 21/01/2017.
Maxime Oquab, Leon Bottou, Ivan Laptev, and Josef Sivic.
Learning and transferring mid-level image representations
using convolutional neural networks. In Proceedings of the
IEEE conference on computer vision and pattern recognition, pages 1717–1724, 2014.
Jeff Orchard and Lin Wang. The evolution of a generalized
neural learning rule. In Neural Networks (IJCNN), 2016
International Joint Conference on, pages 4688–4694. IEEE,
2016.
Jeanne Ellis Ormrod and Kevin M Davis. Human learning.
Merrill, 2004.
Ingo Paenke. Dynamics of Evolution and Learning. PhD thesis, Universität Karlsruhe (TH), Fakultät für Wirtschaftwsissenschaften, 2008.

Ingo Paenke, Bernhard Sendhoff, and Tadeusz J Kawecki.
Influence of plasticity and learning on evolution under
directional selection. The American Naturalist, 170(2):E47–
E58, 2007.
Ingo Paenke, T Kawecki, and Bernhard Sendhoff. The influence of learning on evolution: A mathematical framework.
Artificial Life, 15(2):227–245, 2009.
Sinno Jialin Pan and Qiang Yang. A survey on transfer learning. IEEE Transactions on knowledge and data engineering,
22(10):1345–1359, 2010.
Alvaro Pascual-Leone, Amir Amedi, Felipe Fregni, and Lotfi B
Merabet. The plastic human brain cortex. Annu. Rev.
Neurosci., 28:377–401, 2005.
I. P. Pavlov. Conditioned reflexes. Oxford : Oxford University
Press, 1927.
Cengiz Pehlevan, Tao Hu, and Dmitri B Chklovskii. A
Hebbian/anti-Hebbian neural network for linear subspace
learning: A derivation from multidimensional scaling of
streaming data. Neural computation, 2015.
C. M. A. Pennartz. Reinforcement Learning by Hebbian
Synapses with Adaptive Threshold. Neuroscience, 81(2):
303–319, 1997.
Massimo Pigliucci. Is evolvability evolvable? Nature Reviews
Genetics, 9(1):75–82, 2008.
Justin Pugh, Andrea Soltoggio, and Kenneth O. Stanley. Realtime Hebbian Learning from Autoencoder Features for
Control Tasks. In Proceedings of the Fourteenth International Conference on the Synthesis and Simulation of Living
Systems, 2014.
Josef P. Rauschecker and Wolf Singer. The effects of early
visual experience on the cat’s visual cortex and their possible explanation by Hebb synapses. Journal of Physiology,
310:215–239, 1981.
Aditya Rawal and Risto Miikkulainen. Evolving Deep LSTMbased Memory Networks Using an Information Maximization Objective. In Proceedings of the Genetic and Evolutionary Computation Conference 2016, GECCO ’16, pages
501–508, New York, NY, USA, 2016. ACM. ISBN 978-14503-4206-3.
E. Real, S. Moore, A. Selle, S. Saxena, Y. L. Suematsu, Q. Le,
and A. Kurakin. Large-Scale Evolution of Image Classifiers.
ArXiv e-prints, March 2017.
Scott Reed and Nando de Freitas. Neural programmerinterpreters. arXiv preprint arXiv:1511.06279, 2015.
Robert A Rescorla. Pavlovian Second-Order Conditioning
(Psychology Revivals): Studies in Associative Learning.
Psychology Press, 2014.
Sebastian Risi and Kenneth O. Stanley. Indirectly Encoding
Neural Plasticity as a Pattern of Local Rules. In Proceedings
of the 11th International Conference on Simulation of
Adaptive Behavior (SAB 2010), New York, 2010. Springer.
Sebastian Risi and Kenneth O Stanley. A unified approach to
evolving plasticity and neural geometry. In Neural Networks
(IJCNN), The 2012 International Joint Conference on, pages
1–8. IEEE, 2012.

22

Sebastian Risi and Kenneth O. Stanley.
Guided SelfOrganization in Indirectly Encoded and Evolving Topographic Maps. In Proceedings of the Genetic and Evolutionary Computation Conference (GECCO-2014). New York,
NY: ACM (8 pages), 2014.
Sebastian Risi, Sandy D Vanderbleek, Charles E Hughes,
and Kenneth O Stanley. How novelty search escapes the
deceptive trap of learning to learn. In Proceedings of
the 11th Annual conference on Genetic and evolutionary
computation, pages 153–160. ACM, 2009.
Sebastian Risi, Charles E Hughes, and Kenneth O Stanley.
Evolving plastic neural networks with novelty search. Adaptive Behavior, 18(6):470–491, 2010.
Alan Roberts, Deborah Conte, Mike Hull, Robert MerrisonHort, Abul Kalam al Azad, Edgar Buhl, Roman Borisyuk,
and Stephen R Soffe. Can simple rules control development of a pioneer vertebrate neuronal network generating
behavior? Journal of Neuroscience, 34(2):608–621, 2014.
Anthony Robins. Catastrophic forgetting, rehearsal, and
pseudorehearsal. Connection Science: Journal of Neural
Computing, Artificial Intelligence and Cognitive Research,
7(123-146), 1995.
Derek Roff. Evolution of life histories: theory and analysis.
Springer Science & Business Media, 1993.
Edmund T Rolls and Simon M Stringer. On the design of
neural networks in the brain by genetic evolution. Progress
in Neurobiology, 61(6):557–579, 2000.
David E Rumelhart, Geoffrey E Hinton, and Ronald J
Williams. Learning representations by back-propagating
errors. Cognitive modeling, 5(3):1, 1988.
Stuart Russell and Peter Norvig. Artificial intelligence: a
modern approach. Pearson; 3 edition (5 Aug. 2013), 2013.
Scott J Russo, David M Dietz, Dani Dumitriu, John H Morrison, Robert C Malenka, and Eric J Nestler. The addicted
synapse: mechanisms of synaptic and structural plasticity
in nucleus accumbens. Trends in neurosciences, 33(6):267–
276, 2010.
Andrei A Rusu, Neil C Rabinowitz, Guillaume Desjardins, Hubert Soyer, James Kirkpatrick, Koray Kavukcuoglu, Razvan
Pascanu, and Raia Hadsell. Progressive Neural Networks.
arXiv preprint arXiv:1606.04671, 2016.
Eduardo Sanchez, Daniel Mange, Moshe Sipper, Marco
Tomassini, Andrés Pérez-Uribe, and André Stauffer. Phylogeny, ontogeny, and epigenesis: Three sources of biological inspiration for softening hardware. In International
Conference on Evolvable Systems, pages 33–54. Springer,
1996.
Jürgen Schmidhuber. Curious model-building control systems.
In Neural Networks, 1991. 1991 IEEE International Joint
Conference on, pages 1458–1463. IEEE, 1991.
Jürgen Schmidhuber. Developmental robotics, optimal artificial curiosity, creativity, music, and the fine arts. Connection
Science, 18(2):173–187, 2006.
Jürgen Schmidhuber. Deep learning in neural networks: An
overview. Neural Networks, 61:85–117, 2015.
Jürgen Schmidhuber, Daan Wierstra, Matteo Gagliolo, and
Faustino Gomez. Training recurrent networks by evolino.
Neural computation, 19(3):757–779, 2007.

Christiane Schreiweis, Ulrich Bornschein, Eric Burguière,
Cemil Kerimoglu, Sven Schreiter, Michael Dannemann,
Shubhi Goyal, Ellis Rea, Catherine A French, Rathi
Puliyadi, et al. Humanized Foxp2 accelerates learning
by enhancing transitions from declarative to procedural
performance. Proceedings of the National Academy of
Sciences, 111(39):14253–14258, 2014.
Henning Schroll and Fred H Hamker. Computational models
of basal-ganglia pathway functions: focus on functional
neuroanatomy. Frontiers in systems neuroscience, 7, 2015.
Wolfram Schultz. Predictive Reward Signal of Dopamine
Neurons. Journal of Neurophysiology, 80:1–27, 1998.
Wolfram Schultz, Paul Apicella, and Tomas Ljungberg. Responses of Monkey Dopamine Neurons to Reward and
Conditioned Stimuli during Successive Steps of Learning
a Delayed Response Task. The Journal of Neuroscience,
13:900–913, 1993.
Wolfram Schultz, Peter Dayan, and P. Read Montague. A
Neural Substrate for Prediction and Reward. Science, 275:
1593–1598, 1997.
Fernando Silva, Paulo Urbano, and Anders Lyhne Christensen.
Adaptation of robot behaviour through online evolution and
neuromodulated learning. In Ibero-American Conference on
Artificial Intelligence, pages 300–309. Springer, 2012a.
Fernando Silva, Paulo Urbano, and Anders Lyhne Christensen. Continuous adaptation of robot behaviour through
online evolution and neuromodulated learning. In Fifth
International Workshop on Evolutionary and Reinforcement
Learning for Autonomous Robot Systems (ERLARS 2012) in
conjunction with the 20th European Conference on Artificial
Intelligence (ECAI 2012). Montpellier. Citeseer, 2012b.
David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser,
Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot,
et al. Mastering the game of Go with deep neural networks
and tree search. Nature, 529(7587):484–489, 2016.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv
preprint arXiv:1409.1556, 2014.
Karl Sims. Artificial evolution for computer graphics, volume 25. ACM, 1991.
Karl Sims. Evolving 3D morphology and behavior by competition. Artificial life, 1(4):353–372, 1994.
M. Sipper, E. Sanchez, D. Mange, M. Tomassini, A. PerezUribe, and A. Stauffer. A phylogenetic, ontogenetic, and
epigenetic view of bio-inspired hardware system. Evolutionary Computation, IEEE Transactions on, 1(1):83–97, April
1997.
B. F. Skinner. Science and Human Behavior. New York,
MacMillan, 1953.
Burrhus Frederic Skinner. The behavior of organisms: An
experimental analysis. New York, London, D. AppletonCentury Company, Incorporated, 1938.
John Maynard Smith. When learning guides evolution. Nature,
329(6142):761–762, 1986.

23

T.M.C. Smith, P. Husbands, A. Philippides, and Michael
O’Shea. Neuronal Plasticity and Temporal Adaptivity:
GasNet Robot Control Networks. Adaptive Behavior, 10:
161–183, 2002.
Tom Smith. The evolvability of artificial neural networks for
robot control. PhD thesis, University of Sussex, October
2002.
Andrea Soltoggio. Neural Plasticity and Minimal Topologies
for Reward-based Learning Problems. In Proceeding of the
8th International Conference on Hybrid Intelligent Systems
(HIS2008), 10-12 September, Barcelona, Spain, 2008a.
Andrea Soltoggio. Neuromodulation Increases Decision Speed
in Dynamic Environments. In Proceedings of the 8th International Conference on Epigenetic Robotics, Southampton,
July 2008, 2008b.
Andrea Soltoggio. Evolutionary and Computational Advantages of Neuromodulated Plasticity. PhD thesis, School of
Computer Science, The University of Birmingham, 2008c.
Andrea Soltoggio. Short-term plasticity as cause-effect hypothesis testing in distal reward learing. Biological Cybernetics, 109(1):75–94, 2015.
Andrea Soltoggio and Ben Jones. Novelty of behaviour as a
basis for the neuro-evolution of operant reward learning. In
Proceedings of the 11th Annual conference on Genetic and
evolutionary computation, pages 169–176. ACM, 2009.
Andrea Soltoggio and Kenneth O. Stanley. From Modulated
Hebbian Plasticity to Simple Behavior Learning through
Noise and Weight Saturation. Neural Networks, 34:28–41,
October 2012.
Andrea Soltoggio, Peter Dürr, Claudio Mattiussi, and Dario
Floreano. Evolving Neuromodulatory Topologies for Reinforcement Learning-like Problems. In Proceedings of the
IEEE Congress on Evolutionary Computation, CEC 2007,
2007.
Andrea Soltoggio, John A. Bullinaria, Claudio Mattiussi, Peter Dürr, and Dario Floreano. Evolutionary Advantages
of Neuromodulated Plasticity in Dynamic, Reward-based
Scenarios. In Artificial Life XI: Proceedings of the Eleventh
International Conference on the Simulation and Synthesis
of Living Systems. MIT Press, 2008.
Olaf Sporns and William H. Alexander. Neuromodulation
in a learning robot: interactions between neural plasticity
and behavior. In Proceedings of the International Joint
Conference on Neural Networks, volume 4, pages 2789–
2794, 2003.
Nitish Srivastava, Geoffrey E Hinton, Alex Krizhevsky, Ilya
Sutskever, and Ruslan Salakhutdinov. Dropout: a simple
way to prevent neural networks from overfitting. Journal of
Machine Learning Research, 15(1):1929–1958, 2014.
J. E. R. Staddon. Adaptive Behaviour and Learning. Cambridge University Press, 1983.
Kenneth O Stanley. Compositional pattern producing networks: A novel abstraction of development. Genetic programming and evolvable machines, 8(2):131–162, 2007.
Kenneth O Stanley and Joel Lehman. Why Greatness Cannot
Be Planned. Springer Science Business Media., 10:978–3,
2015.

Kenneth O Stanley and Risto Miikkulainen. Evolving neural
networks through augmenting topologies. Evolutionary
computation, 10(2):99–127, 2002.
Kenneth O Stanley and Risto Miikkulainen. A taxonomy for
artificial embryogeny. Artificial Life, 9(2):93–130, 2003a.
Kenneth O. Stanley and Risto Miikkulainen. Evolving Adaptive Neural Networks with and without Adaptive Synapses.
In Bart Rylander, editor, Genetic and Evolutionary Computation Conference Late Breaking Papers, pages 275–282,
Chicago, USA, 12–16July 2003b.
Kenneth O Stanley, David B D’Ambrosio, and Jason Gauci.
A hypercube-based encoding for evolving large-scale neural
networks. Artificial life, 15(2):185–212, 2009.
Luc Steels. The artificial life roots of artificial intelligence.
Artificial life, 1(1 2):75–110, 1993.
James V Stone. Distributed representations accelerate evolution of adaptive behaviours. PLoS computational biology,
3(8):e147, 2007.
R. E. Suri. TD models of reward predictive responses in
dopamine neurons. Neural Networks, 15:523–533, 2002.
R. E. Suri, J. Bargas, and M. A. Arbib. Modeling functions
of striatal dopamine modulation in learning and planning.
Neuroscience, 103(1):65–85, 2001.
Richard S. Sutton and Andrew G. Barto. Reinforcement
Learning: An Introduction. MIT Press, Cambridge, MA,
USA, 1998.
Tim Taylor, Mark Bedau, Alastair Channon, David Ackley,
Wolfgang Banzhaf, Guillaume Beslon, Emily Dolson, Tom
Froese, Simon Hickinbotham, Takashi Ikegami, et al. Openended evolution: perspectives from the OEE workshop in
York. Artificial life, 2016.
Marc Tessier-Lavigne and Corey S Goodman. The molecular
biology of axon guidance. Science, 274(5290):1123, 1996.
Edward Lee Thorndike. Animal Intelligence. New York:
Macmillan, 1911.
Sebastian Thrun and Tom M Mitchell. Lifelong robot learning.
Robotics and autonomous systems, 15(1-2):25–46, 1995.
Sebastian Thrun and Joseph O’Sullivan. Discovering structure
in multiple learning tasks: The TC algorithm. In ICML,
volume 96, pages 489–497, 1996.
Sebastian Thrun and Lorien Pratt. Learning to learn. Springer
Science & Business Media, 2012.
Paul Tonelli and Jean-Baptiste Mouret. Using a map-based
encoding to evolve plastic neural networks. In Evolving and
Adaptive Intelligent Systems (EAIS), 2011 IEEE Workshop
on, pages 9–16. IEEE, 2011.
Paul Tonelli and Jean-Baptiste Mouret. On the relationships
between generative encodings, regularity, and learning abilities when evolving plastic artificial neural networks. PloS
one, 8(11):e79138, 2013.
J. Urzelai and D Floreano. Evolutionary Robotics: Coping
with Environmental Change. In Genetic and Evolutionary
Computation Conference (GECCO’2000), 2000.
Roby Velez and Jeff Clune. Diffusion-based neuromodulation can eliminate catastrophic forgetting in simple neural
networks. CoRR, abs/1705.07241, 2017.

24

Santosh S Venkatesh. Directed drift: A new linear threshold
algorithm for learning binary weights on-line. Journal of
Computer and System Sciences, 46(2):198–217, 1993.
Julien Vitay and Fred H. Hamker. A computational model of
basal ganglia and its role in memory retrieval in rewarded
visual memory tasks. Frontiers in Computational Neuroscience, 2010.
Günter P Wagner and Lee Altenberg. Perspective: complex
adaptations and the evolution of evolvability. Evolution, 50
(3):967–976, 1996.
Edgar T. Walters and John H. Byrne. Associative Conditioning
of Single Sensory Neurons Suggests a Cellular Mechanism
for Learning. Science, 219:405–408, 1983.
Bernard Widrow and Michael A Lehr. 30 years of adaptive
neural networks: perceptron, madaline, and backpropagation. Proceedings of the IEEE, 78(9):1415–1442, 1990.
Bernard Widrow, Marcian E Hoff, et al. Adaptive switching
circuits. In IRE WESCON convention record, volume 4,
pages 96–104. New York, 1960.
David Willshaw and Peter Dayan. Optimal plasticity from
matrix memories: What goes up must come down. Neural
Computation, 2(1):85–93, 1990.

Brian M. Yamauchi and Randall D. Beer. Sequential Behavior
and Learning in Evolved Dynamical Neural Networks.
Adaptive Behavior, 2(3), 1994.
Xin Yao. Evolving Artificial Neural Networks. Evolutionary Computation, IEEE Transactions on, 87(9):1423–1447,
September 1999.
Xin Yao and Yong Liu. A new evolutionary system for
evolving artificial neural networks. IEEE transactions on
neural networks, 8(3):694–713, 1997.
Jason Yoder and Larry Yaeger. Evaluating Topological Models
of Neuromodulation in Polyworld. In Artificial Life XIV,
MIT Press, Cambridge, MA, 2014.
Steven R Young, Derek C Rose, Thomas P Karnowski, SeungHwan Lim, and Robert M Patton. Optimizing deep learning
hyper-parameters through an evolutionary algorithm. In
Proceedings of the Workshop on Machine Learning in HighPerformance Computing Environments, page 4. ACM, 2015.
Tom Ziemke and Mikael Thieme.
Neuromodulation of
Reactive Sensorimotor Mappings as Short-Term Memory
Mechanism in Delayed Response Tasks. Adaptive Behavior,
10:185–199, 2002.

