Nature vs. Nurture: The Role of Environmental Resources in Evolutionary Deep
Intelligence

arXiv:1802.03318v1 [] 9 Feb 2018

Audrey G. Chung, Paul Fieguth, and Alexander Wong
Vision & Image Processing Research Group
Dept. of Systems Design Engineering, University of Waterloo
Waterloo, ON, Canada, N2L 3G1
Emails: {agchung, pfieguth, a28wong}@uwaterloo.ca

Abstract—Evolutionary deep intelligence synthesizes highly
efficient deep neural networks architectures over successive
generations. Inspired by the nature versus nurture debate, we
propose a study to examine the role of external factors on
the network synthesis process by varying the availability of
simulated environmental resources. Experimental results were
obtained for networks synthesized via asexual evolutionary synthesis (1-parent) and sexual evolutionary synthesis (2-parent,
3-parent, and 5-parent) using a 10% subset of the MNIST
dataset. Results show that a lower environmental factor model
resulted in a more gradual loss in performance accuracy and
decrease in storage size. This potentially allows significantly
reduced storage size with minimal to no drop in performance
accuracy, and the best networks were synthesized using the
lowest environmental factor models.
Keywords-Deep neural networks; deep learning; evolutionary deep intelligence; evolutionary synthesis; environmental
resource models

I. I NTRODUCTION
In recent years, deep neural networks [1]–[4] have experienced an explosion in popularity due to their ability
to accurately represent complex data and their improved
performance over other state-of-the-art machine learning
methods. This notable increase in performance has mainly
been attributed to increasingly large deep neural network
model sizes and expansive training datasets, resulting in
growing computational and memory requirements [5].
For many practical scenarios, however, these computational requirements make powerful deep neural networks
infeasible. Applications such as self-driving cars and consumer electronics are often limited to low-power embedded
GPUs or CPUs, making compact and highly efficient deep
neural networks very desirable. While methods for directly
compressing large neural network models into smaller representations have been developed [5]–[10], Shafiee et al. [11]
proposed a radically novel approach: Can deep neural networks naturally evolve to be highly efficient?
Taking inspiration from nature, Shafiee et al. introduced
the concept of evolutionary deep intelligence to organically
synthesize increasingly efficient and compact deep neural
networks over successive generations. Evolutionary deep intelligence mimics biological evolutionary mechanisms found

in nature using three computational constructs: i) heredity,
ii) natural selection, and iii) random mutation.
While the idea of utilizing evolutionary techniques to
generate and train neural networks has previously been
explored [12]–[16], evolutionary deep intelligence [11] introduces some key differences. In particular, where past
works use classical evolutionary computation methods such
as genetic algorithms, [11] introduced a novel probabilistic
framework that models network genetic encoding and external environmental conditions via probability distributions.
Additionally, these previous studies have been primarily
focused on improving a deep neural network’s performance
accuracy, while evolutionary deep intelligence shifts part of
the focus to synthesizing highly efficient neural network
architectures while maintaining high performance accuracy.
Following the seminal evolutionary deep intelligence paper [11], Shafiee et al. [17] further proposed a detailed
extension of the original approach via synaptic clusterdriven genetic encoding. This work introduced a multi-factor
synapse probability model, and comprehensive experimental
results using four well-known deep neural networks [18]
produced significantly more efficient network architectures
specifically tailored for GPU-accelerated applications while
maintaining state-of-the-art performance accuracy. Additional research has since been conducted to include a synaptic precision constraint during the evolutionary synthesis
of network architectures [19] and to introduce the concept
of trans-generational genetic transmission of environmental
information [20].
Shafiee et al.’s work [11], [17], however, formulated the
evolutionary synthesis process based on asexual reproduction. Recently, Chung et al. [21], [22] extended the asexual
evolutionary sythensis approach used previously in [11], [17]
to m-parent sexual evolutionary synthesis, demonstrating
that increasing the number of parent networks resulted in
synthesizing networks with improved architectural efficiency
with only a 2 − 3% drop in performance accuracy.
In addition to demonstrating state-of-the-art performance
in classic computer vision tasks, evolutionary deep intelligence has proven promising in a variety of applications
such as real-time motion detection in videos on embedded

Figure 1: A visualization of the m-parent evolutionary synthesis process over successive generations. In this study, we examine
the role of simulated external environmental factors (indicated by red arrows) during the m-parent network synthesis process
over successive generations.

systems [23], imaging-based lung cancer detection via an
evolutionary deep radiomic sequencer discovery approach
using clinical lung CT images with pathologically proven
diagnostic data [24], and synthesizing even smaller deep
neural architectures based on the more recent SqueezeNet
v1.1 macroarchitecture for applications with fewer target
classes [25]. Evolutionary deep intelligence enables the use
of powerful deep neural networks in applications where
limited computational power, limited memory resources,
and/or privacy are of concern.
The effect of environmental factors (e.g., abundance or
scarcity of resources) during the evolutionary synthesis
process on generations of synthesized network architectures
has largely been unexplored. In this study, we aim to
better understand the role these environmental resources
models play by varying the availability of simulated environmental resources in m-parent evolutionary synthesis.
The evolutionary deep intelligence method and the proposed
environmental resource study are described in Section II.
The experimental setup and results are presented in Section III. Lastly, conclusions and future work are discussed
in Section IV.
II. M ETHODS
We propose a study to determine the role of simulated
environmental resources during the evolutionary synthesis
process (Figure 1) and their effects on generations of synthesized neural networks. In this work, we leverage the
evolutionary deep intelligence framework (as formulated
below), using previously proposed cluster-driven genetic
encoding [17] and m-parent evolutionary synthesis [22]
methods, and vary the availability of simulated external
environmental resources.

A. Sexual Evolutionary Synthesis
Let the network architecture be formulated as H(N, S),
where N denotes the set of possible neurons and S the set
of possible synapses in the network. Each neuron nj ∈ N
is connected to neuron nk ∈ N via a set of synapses s̄ ⊂ S,
such that the synaptic connectivity sj ∈ S has an associated
wj ∈ W to denote the connection’s strength. In Shafiee et
al.’s seminal paper on evolutionary deep intelligence [11],
the synthesis probability P (Hg |Hg−1 , Rg ) of a synthesized
network at generation g is approximated by the synaptic
probability P (Sg |Wg−1 , Rg ); this emulates heredity through
the generations of networks, and is also conditional on an
environmental factor model Rg to imitate the availability of
resources in the network’s environment. Thus, the synthesis
probability can be modelled as follows:
P (Hg |Hg−1 , Rg ) ' P (Sg |Wg−1 , Rg ).

(1)

Introducing the synaptic cluster-driven genetic encoding
approach, Shafiee et al. [17] proposed that the synthesis
probability incorporate a multi-factor synaptic probability
model and different quantitative environmental factor models
at the synapse and cluster levels:
P (Hg |Hg−1 , Rg ) =
i
Yh
Y
P (sg,c |Wg−1 , Rcg ) ·
P (sg,j |wg−1,j , Rsg )
j∈c

c∈C

(2)
Rcg

Rsg

where
and
represent the environmental factor models enforced during synthesis at the cluster level and the
synapse level, respectively. P (sg,c |Wg−1 , Rcg ) represents the
probability of synthesis for a given cluster of synapses

sg,c . Thus, P (sg,c |Wg−1 , Rcg ) denotes the likelihood that
a synaptic cluster sg,c (for all clusters c ∈ C) will exist in
the network architecture in generation g given the cluster’s
synaptic strength in generation g−1 and the cluster-level environmental factor model. Comparably, P (sg,j |wg−1,j , Rsg )
represents the likelihood of the existence of synapse j within
the synaptic cluster c in generation g given the synaptic
strength in the previous generation g − 1 and synapse-level
environmental factor model. This multi-factor probability
model encourages both the persistence of strong synaptic
clusters and the persistence of strong synaptic connectivity
over successive generations [17].
Chung et al. [22] proposed a further modification of the
synthesis probability P (Hg |Hg−1 , Rg ) via the incorporation
of a m-parent synthesis process to drive network diversity
and adaptability by mimicking sexual reproduction. The synthesis probability was reformulated to combine the cluster
and synapse probabilities of m parent networks via some
cluster-level mating function Mc (·) and some synapse-level
mating function Ms (·):
P (Hg(i) |HGi , Rg(i) ) =
Yh
P (sg(i),c |Mc (WHGi ), Rcg(i) )·
c∈C

Y

i
P (sg(i),j |Ms (wHGi ,j ), Rsg(i) ) .

(3)

j∈c

B. Mating Rituals of Deep Neural Networks
In the context of this study, we restrict HGi to networks
in the immediately preceding generation, i.e., for a newly
synthesized network Hg(i) at generation g(i), the m parent
networks in HGi are from generation g(i) − 1. As in [22],
the mating functions are:
Mc (WHGi ) =
Ms (wHGi ,j ) =

m
X
k=1
m
X

αc,k WHk

(4)

αs,k wHk ,j

(5)

resources available (Rcg(i) ), and γ is a randomly generated
number between 0 and 1.
Previous studies in evolutionary deep intelligence [11],
[17], [18], [21], [22] generally employed a environmental
factor model of Rcg(i) = 70, i.e., the probability of existing
clusters being stochastically dropped during the evolutionary
synthesis process was scaled by 70%. To better understand
the role environmental resources play, we propose a study
varying the cluster-level environmental factor model Rcg(i) .
III. R ESULTS
A. Experimental Setup
In this study, the m-parent evolutionary synthesis of deep
neural networks were performed over multiple generations,
and the effects of various environmental factor models were
explored using 10% of the MNIST [26] hand-written digits
dataset with the first generation ancestor networks trained
using the LeNet-5 architecture [27]. Figure 2 shows sample
images from the MNIST dataset.
Similar to Shafiee et al.’s work [17], each filter (i.e.,
collection of kernels) was considered as a synaptic cluster in
the multi-factor synapse probability model. In this work, we
assessed the synthesized networks using performance accuracy on the MNIST dataset and storage size (representative
of the architectural efficiency of a network) of the networks
with respect to the computational time required.
To investigate the effects the cluster-level environmental
factor model Rcg(i) , we vary Rcg(i) from 50% to 95% at 5%
increments, i.e.,:
Rcg(i) = {50, 55, 60, 65, 70, 75, 80, 85, 90, 95}

(7)

As in [17], we use a synapse-level environmental factor
model of Rsg(i) = 70% to allow for increasingly more
compact and efficient network architectures in the successive
generations while minimizing any loss in accuracy.

k=1

where WHk represents the cluster’s synaptic strength for the
k th parent network Hk ∈ HGi . Similarly, wHk ,j represents
the synaptic strength of a synapse j within cluster c for the
k th parent network Hk ∈ HGi .
C. The Role of Environmental Resources
In this work, we aim to study the effects of the clusterlevel environmental factor model Rcg(i) . At the cluster level,
the existence of all clusters c ∈ C is determined as:
(
1 Rcg(i) · (1 − |WHGi |) ≤ γ
c
1g(i) =
(6)
0 otherwise
where 1cg(i) incorporates both the strength of the synapses
in a cluster (WHGi ) and the cluster-level environmental

Figure 2: Sample images from the MNIST hand-written
digits dataset [26].

Figure 3: Performance accuracy (left) and storage size (right) with respect to computational time for 1-parent evolutionary
synthesis using various environmental factor models. Networks synthesized via 1-parent evolutionary synthesis and subject
to various environmental factors show a clear monotonic trend.

B. Experimental Results
Figure 3 shows the performance accuracy and storage size
for 1-parent evolutionary synthesis using various clusterlevel environmental factor models, and shows a clear monotonic trend with respect to the environmental factors. As the
environmental factor model decreases, the rate of decrease
in storage size of synthesized networks slows accordingly
and the rate of performance accuracy loss over generations
is similarly slowed. As such, Figure 3 shows that networks
synthesized with an environmental factor of 50% (dark blue)
has the most gradual decrease in performance accuracy and
storage size, while networks synthesized with an environmental factor of 90% (light purple) or 95% (dark purple)
have the steepest decrease in performance accuracy and
storage size over successive generations.
One aspect to note in Figure 3 is the bottom plateau in
performance accuracy at 10%; this is due to the MNIST
dataset [26] itself which consists of 10 classes of individual handwritten digits (i.e., digits 0 to 9), and the 10%
performance accuracy is akin to random guessing. There
is similarly a bottom plateau in network storage size that
is most easily seen in the trends for environmental factors
of 90% (light purple) and 95% (dark purple). This plateau
corresponds to networks where there is only a single synapse
in each convolution layer and, thus, cannot be reduced any
further. This network storage size reduction of approximately three orders of magnitude is a result of using the
LeNet-5 [27] architecture as the first generation ancestor
network, and there is no inherent limit in network reduction
when using the evolutionary deep intelligence approach.
The performance accuracy and storage size for 2-parent,
3-parent, and 5-parent evolutionary synthesis using various

cluster-level environmental factor models (Figure 4) show
similar general trends to 1-parent evolutionary synthesis
in Figure 3. There is noticeably more variability in both
performance accuracy and storage size as the number of
parent networks increases; this is likely a result of only
one synthesized network being represented per generation
in Figure 4. Lastly, while there is synaptic competition as
imposed by the environmental factor models, the network
synthesis process is non-competitive between parent networks (i.e., no selection criteria) and potentially contributes
to the variability in performance accuracy and storage size
over successive generations.
Figure 5 shows performance accuracy as a function of
storage size for 1-parent asexual evolutionary synthesis (left)
and m-parent sexual evolutionary synthesis (right) using
various cluster-level environmental factor models, where the
best synthesized networks are closest to the top left corner,
i.e., high performance accuracy and low storage size. In
the m-parent sexual evolutionary synthesis figure, 2-parent
evolutionary synthesis is represented using the smallest
points, 3-parent evolutionary synthesis is represented by
medium-sized points, and 5-parent evolutionary synthesis is
represented using the largest points.
Notice that in 1-parent evolutionary synthesis, most of
the networks closest to the top left corner are synthesized
using environmental factors of 50%, 55%, and 60%. Similarly for m-parent evolutionary synthesis, the best networks
(regardless of number of parents) are generally synthesized
using the lowest environmental factor models. Specifically,
environmental factor models of 50% (dark blue), 55% (light
blue), and 60% (dark green) produced the majority of the
synthesized networks closest to the top left corner.

Figure 4: Performance accuracy (left) and storage size (right) with respect to computational time for 2-parent evolutionary
synthesis (top row), 3-parent evolutionary synthesis (middle row), and 5-parent evolutionary synthesis (bottom row) using
various environmental factor models.

Figure 5: Performance accuracy as a function of storage size for 1-parent asexual evolutionary synthesis (left) and m-parent
sexual evolutionary synthesis (right) using various environmental factor models. 2-parent evolutionary synthesis (smallest
points), 3-parent evolutionary synthesis (medium points), and 5-parent evolutionary synthesis (largest points) show noticeably
more variation than the 1-parent case.

IV. C ONCLUSION
In this work, we examined the role external environmental
resource models played during the evolutionary synthesis
process. To achieve this, we varied the availability of simulated environmental resources (using environmental factor
models of 50% to 95% at 5% increments) in the context
of m-parent evolutionary synthesis for m = 1, 2, 3, 5 over
successive generations, and analysed the resulting trends in
performance accuracy and network storage size.
Overall, a lower environmental factor model resulted in a
more gradual loss in performance accuracy and decrease in
storage size. This potentially allows for significant reduction
in storage size with minimal drop in performance accuracy
(e.g., environmental factor 50% in 5-parent evolutionary synthesis). In addition, the best networks were synthesized using
the lowest environmental factor models, i.e., environmental
factors of 50%, 55%, and 60%. However, there is a tradeoff
between decreasing the environmental factor model (and,
as a result, the rate at which newly synthesized networks
drop synapses) and increasing network training time due to
a higher number of synapse (i.e., trainable parameters).
Future work includes the investigation of a time-varying
environmental factor models (e.g., increasing or decreasing
the available resources over generations) to imitate changing
environmental conditions, and the incorporation of parent
network selection criteria to mimic natural selection in
the face of scarce environmental resources. In addition,
further analysis of the variability in performance accuracy
and storage size over successive generations is necessary,
particularly with increasing the number of parent networks
during evolutionary synthesis.

ACKNOWLEDGEMENT
This work was supported by the Natural Sciences and Engineering Research Council of Canada and Canada Research
Chairs Program. The authors also thank Nvidia for the GPU
hardware used in this study through the Nvidia Hardware
Grant Program.
R EFERENCES
[1] Y. Bengio, “Learning deep architectures for AI,” Foundations
and trends in Machine Learning, vol. 2, no. 1, pp. 1–127,
2009.
[2] A. Graves, A.-r. Mohamed, and G. Hinton, “Speech recognition with deep recurrent neural networks,” in 2013 IEEE
international conference on acoustics, speech and signal
processing. IEEE, 2013, pp. 6645–6649.
[3] Y. LeCun, Y. Bengio, and G. Hinton, “Deep learning,” Nature,
vol. 521, no. 7553, pp. 436–444, 2015.
[4] J. J. Tompson, A. Jain, Y. LeCun, and C. Bregler, “Joint
training of a convolutional network and a graphical model for
human pose estimation,” in Adv ances in neural information
processing systems, 2014, pp. 1799–1807.
[5] S. Han, J. Pool, J. Tran, and W. Dally, “Learning both weights
and connections for efficient neural network,” in Advances
in Neural Information Processing Systems, 2015, pp. 1135–
1143.
[6] Y. LeCun, J. S. Denker, S. A. Solla, R. E. Howard, and L. D.
Jackel, “Optimal brain damage,” in NIPs, vol. 2, 1989, pp.
598–605.
[7] Y. Gong, L. Liu, M. Yang, and L. Bourdev, “Compressing
deep convolutional networks using vector quantization,” arXiv
preprint arXiv:1412.6115, 2014.

[8] S. Han, H. Mao, and W. J. Dally, “Deep compression:
Compressing deep neural network with pruning, trained quantization and huffman coding,” CoRR, abs/1510.00149, vol. 2,
2015.
[9] W. Chen, J. T. Wilson, S. Tyree, K. Q. Weinberger, and
Y. Chen, “Compressing neural networks with the hashing
trick,” CoRR, abs/1504.04788, 2015.
[10] Y. Sun, X. Wang, and X. Tang, “Sparsifying neural network
connections for face recognition,” in Proceedings of the IEEE
Conference on Computer Vision and Pattern Recognition,
2016, pp. 4856–4864.
[11] M. J. Shafiee, A. Mishra, and A. Wong, “Deep learning with
Darwin: Evolutionary synthesis of deep neural networks,”
Neural Processing Letters, 2017.
[12] P. J. Angeline, G. M. Saunders, and J. B. Pollack, “An evolutionary algorithm that constructs recurrent neural networks,”
IEEE transactions on Neural Networks, vol. 5, no. 1, pp. 54–
65, 1994.
[13] J. Gauci and K. Stanley, “Generating large-scale neural networks through discovering geometric regularities,” in Proceedings of the 9th annual conference on Genetic and evolutionary computation. ACM, 2007, pp. 997–1004.
[14] K. O. Stanley, B. D. Bryant, and R. Miikkulainen, “Real-time
neuroevolution in the NERO video game,” IEEE transactions
on evolutionary computation, vol. 9, no. 6, pp. 653–668, 2005.
[15] K. O. Stanley and R. Miikkulainen, “Evolving neural networks through augmenting topologies,” Evolutionary computation, vol. 10, no. 2, pp. 99–127, 2002.
[16] S. S. Tirumala, S. Ali, and C. P. Ramesh, “Evolving deep
neural networks: A new prospect,” in Natural Computation,
Fuzzy Systems and Knowledge Discovery (ICNC-FSKD),
2016 12th International Conference on. IEEE, 2016, pp.
69–74.
[17] M. J. Shafiee and A. Wong, “Evolutionary synthesis of deep
neural networks via synaptic cluster-driven genetic encoding,”
in Advances in Neural Information Processing Systems, 2016.
[18] M. J. Shafiee, E. Barshan, and A. Wong, “Evolution in groups:
a deeper look at synaptic cluster driven evolution of deep
neural networks,” arXiv preprint arXiv:1704.02081, 2017.
[19] M. J. Shafiee, F. Li, and A. Wong, “Exploring the imposition
of synaptic precision restrictions for evolutionary synthesis
of deep neural networks,” arXiv preprint arXiv:1707.00095,
2017.
[20] M. J. Shafiee, E. Barshan, F. Li, B. Chwyl, M. Karg, C. Scharfenberger, and A. Wong, “Learning efficient deep feature
representations via transgenerational genetic transmission of
environmental information during evolutionary synthesis of
deep neural networks,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017, pp.
979–986.

[21] A. G. Chung, M. Javad Shafiee, P. Fieguth, and A. Wong,
“The mating rituals of deep neural networks: Learning compact feature representations through sexual evolutionary synthesis,” in Proceedings of the IEEE Conference on Computer
Vision and Pattern Recognition, 2017, pp. 1220–1227.
[22] A. G. Chung, P. Fieguth, and A. Wong, “Polyploidism in
deep neural networks: m-parent evolutionary synthesis of
deep neural networks in varying population sizes,” Journal
of Computational Vision and Imaging Systems, vol. 3, no. 1,
2017.
[23] M. J. Shafiee, P. Siva, P. Fieguth, and A. Wong, “Realtime embedded motion detection via neural response mixture
modeling,” Journal of Signal Processing Systems, pp. 1–16,
2017.
[24] M. J. Shafiee, A. G. Chung, F. Khalvati, M. A. Haider,
and A. Wong, “Discovery radiomics via evolutionary deep
radiomic sequencer discovery for pathologically proven lung
cancer detection,” Journal of Medical Imaging, vol. 4, no. 4,
p. 041305, 2017.
[25] M. J. Shafiee, F. Li, B. Chwyl, and A. Wong, “Squishednets: Squishing squeezenet further for edge device scenarios via deep evolutionary synthesis,” arXiv preprint
arXiv:1711.07459, 2017.
[26] Y. LeCun, C. Cortes, and C. J. Burges, “The MNIST database
of handwritten digits,” 1998.
[27] Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner, “Gradientbased learning applied to document recognition,” Proceedings
of the IEEE, vol. 86, no. 11, pp. 2278–2324, 1998.

