arXiv:1711.03467v1 [cs.NE] 9 Nov 2017

Worm-level Control through Search-based
Reinforcement Learning
Mathias Lechner∗
TU Wien
Austria

Radu Grosu
TU Wien
Austria

Ramin M. Hasani
TU Wien
Austria

Abstract
Through natural evolution, nervous systems of organisms formed near-optimal
structures to express behavior. Here, we propose an effective way to create control agents, by re-purposing the function of biological neural circuit models, to
govern similar real world applications. We model the tap-withdrawal (TW) neural
circuit of the nematode, C. elegans, a circuit responsible for the worm’s reflexive response to external mechanical touch stimulations, and learn its synaptic
and neural parameters as a policy for controlling the inverted pendulum problem.
For reconfiguration of the purpose of the TW neural circuit, we manipulate a
search-based reinforcement learning. We show that our neural policy performs
as good as existing traditional control theory and machine learning approaches.
A video demonstration of the performance of our method can be accessed at
https://youtu.be/o-Ia5IVyff8.

Introduction
The nervous system of the soil-worm, C. elegans, has entirely been mapped, demonstrating a nearoptimal wiring structure [13]. An adult hermaphrodite’s brain is composed of 302 neurons hard-wired
by around 8000 chemical and electrical synapses [2]. Function of many neural circuits within its
brain has been identified [14, 1, 4, 6]. In particular, a neural circuit which is responsible for inducing
a forward/backward locomotion reflex when the worm is mechanically exposed to touch stimulus
on its body, has been well-characterized [1] (see Fig. 1A for an illustration of this reflex). Synaptic
polarities of the circuit have then been predicted, suggesting that the circuit realizes a competitive
behavior between forward and backward reflexes, in presence of touch stimulations [14, 15].
Behavior of the tap-withdrawal (TW) reflexive response is substantially similar to the impulse
response of a controller operating on an Inverted Pendulum [9] dynamic system, as illustrated in Fig.
1A and 1B. We thought of taking advantage of such similarity and reconfigure the synaptic and neural
parameters of a deterministic model of the TW neural circuit, to control the inverted pendulum within
the OpenAI’s Roboschool environment [11], while preserving the near-optimal wiring structure of the
circuit. We deploy a search-based reinforcement learning (RL) algorithm for synaptic parametrization
and discuss the performance of the TW re-purposed network, throughly.

Methods
In this section we first define how the TW circuit interacts with the inverted pendulum environment
and then throughly discuss the parameter optimization of the circuit by a search-based RL algorithm.
∗

Correspondence to mathias.lechner@hotmail.com

31st Conference on Neural Information Processing Systems (NIPS 2017), Long Beach, CA, USA.

A

B

C
REV

PVD

p ))
(( Ta

PLM

AVA

AVD

p ))
(( Ta
PVC

AVB

AVM

ALM

DVA

FWD

Response

Response

Legend:
Sensory neuron

Excitatory synapse

Inter neuron

Inhitory synapse

Motor neuron

Gap Junction

Figure 1: Illustration of the touch withdrawal reflex. A) Touching the worm’s tail will excite the
touch sensory neuron PLM, and correspondingly induces a forward locomotion command in the
animal [1] (Hand is not drawn to scale!). B) Working principle of the introduced touch withdrawal
inverted-pendulum controller. C) Tap Withdrawal neural circuit of C. elegans.

Manipulating the model of the TW neural circuit for the inverted pendulum swingup problem

We first introduce the neuron and synapse models we utilized to make an artificial TW neural
circuit. Neurons are modeled by the well-known, deterministicordinary differential
equation (ODE) ,
 P
(i)
n
dvi
Leaky-integrate-and-fire (LIF) model [3] as Cm dt = GLeak VLeak − vi (t) + i=1 Iin , where
(i)

Cm , GLeak and VLeak are parameters of the neuron and Iin , stands for the external currents to the
cell.

Chemical synapses are where two neurons trade information by the release of neurotransmitters.
Chemical synaptic currents are calculated by a non-linear component standing for their conductance
strength, which exponentially depends on the presynaptic neurons’ potential, Vpre , as g(Vpre ) =
w/1 + eσ(Vpre +µ) . The synapse model adopted from [5]. These currents linearly depend on the
postsynaptic neuron’s membrane potential as well, as Is = g(Vpre )(E − Vpost ), where by varying
E, they can make inhibitory or excitatory connection to their postsynaptic neurons. An electrical
synapse (gap-junction), which is a physical junction between two neurons, is modeled by a constant
conductance model, where based
 on the Ohm’s law their bidirectional current between neurons j and
ˆ
i, can be computed as Ii,j = ω̂ vj (t) − vi (t) .

To simulate a neural circuit we used a fixed step implicit numerical ODE solver [7].

For interacting with environment, We introduce sensory and motor neurons. A sensory component
consists of two neurons Sp , Sn and a measurable dynamic system variable x. Sp gets activated when
x has a positive value, whereas Sn fires when x is negative. Mathematically, the potential of the
2

Environment variable
Type
Positive neuron Negative neuron
ϕ
Observation
PLM
AVM
ϕ̇
Observation
ALM
PVD
a
Action
FWD
REV
Table 1: Mapping of environment variables to sensory and motor neurons

neurons Sp , and Sn , as a function of x, can be expressed as

if x ≤ 0
−70mV
50mV
Sp (x) := −70mV + xmax x if 0 < x ≤ xmax

−20mV
if x > xmax

if x ≥ 0
−70mV
50mV
Sn (x) := −70mV + xmin x if 0 > x ≥ xmin

−20mV
if x < xmin .

(1)

(2)

This maps the region [xmin , xmax ] of system variable x, to a membrane potential range of
[−70mV, −20mV ]. Note that the potential range is selected to be close to the biophysics of the nerve
cells where the resting potential is usually set around -70 mV and a neuron can be considered to be
active when it has a potential around -20 mV [5]. The TW neural circuit shown in Figure (1C) has
four sensory neurons: PVD, PLM, AVA and ALM. It therefore, allows us to map the circuit to two
dynamic system variables. The inverted pendulum environment provides four observation variables:
The position of the cart x, together with its velocity ẋ, the angle of the pendulum ϕ1 along with its
angular velocity ϕ̇. Thus in order to map our TW circuit model to the environment, a compromise
that selects two out of these four observation variables has to be found.
Since the objective of the inverted pendulum task is to control the angle in an upward way, we decided
to feed ϕ and ϕ̇ as the inputs to the TW circuit. Consequently, the controller circuit is not aware of
where the cart is or whether the boundary of the movable space is reached. two input variables are
mapped to the circuit elements as shown in Table 1.
Similar to sensory neurons, a motor component is comprised of two neurons Mn , Mp and a controllable motor variable y. Values of y is computed by y := yp + yn and

if Mp > −20mV
ymax
ymax
(3)
yp (Mp ) := 50mV
(Mp + 70mV ) if − 70mV < Mp ≤ −20mV

0
if Mp < −70mV

if Mn > −20mV
ymin
ymin
yn (Mn ) := 50mV
(4)
(Mn + 70mV ) if − 70mV < M n ≤ −20mV

0
if Mn < −70mV

This maps the neuron potentials Mn and Mp , to the range [ymin , ymax ]. The TW circuit in Fig.
(1C) contains two abstract representation for motor neurons involved in reversal movement as REV,
and the ones functioning in forward locomotion, FWD. These neurons are allocated to the motor
command of the cart in the inverted pendulum environment, as expressed in Table 1.
Search-based reinforcement learning
In this section we formulate a RL setting for training the parameters of the neural circuit to perform
the swingup of the inverted pendulum.
The behavior of a neural circuit can be expressed as a policy πθ (oi , si ) 7→ hai+1 , si+1 i, that maps
an observation oi , and an internal state si , to an action ai+1 , and a new internal state si+1 . This
policy acts upon a possible stochastic environment Env(ai+1 ), that provides an observation oi+1 ,
PT
and reward ri+1 , feedback. The stochastic return is given by
 R(θ) := t=1 rt . Objective of the
Reinforcement learning is to find a θ that maximizes E R(θ) .
1

Remark: The environment further splits ϕ into sin(ϕ) and cos(ϕ) to avoid the 2π → 0 discontinuity

3

Algorithm 1: Random Search with Decaying Objective Indicator
Parameter
Cm
GLeak
Erev excitatory
Erev inhibitory
VLeak
µ
σ
ω
ω̂

Min value
1mF
50mS
0mV
-90mV
-90mV
-40mV
0.05
0S
0S

Max value
1F
5S
0mV
-90mV
0mV
-40mV
0.5
3S
3S

1
2
3
4
5
6
7
8
9

Table 2: Table of parameters and their boundaries of a
neural circuit

10
11
12
13
14
15

Input: A stochastic objective indicator f , a
starting parameter θ
Output: Optimized parameter θ
fθ ← f (θ);
for k ← 1 to maximum iterations do
θ0 ← θ + rand();
fθ0 ← f (θ0 );
if fθ0 < fθ then
Set θ ← θ0 ;
fθ ← fθ 0 ;
i ← 0;
end
i ← i + 1;
if i > N then
fθ ← f (θ);
end
end
return θ;

Approaches to find such optimal θ, can be categorized into two major sections, based on how
randomness is formulated for the environment explorations [10]: I-Gradient-based and II-searchbased methods.
Gradient-based RL performs initial random sampling for generating ai , and uses action’s influence on
the return value, to update θ [16]. Search-based methods randomly sample parameters and evaluate
how good they perform to update θ [10]. Here, we adopt a search-based optimization as these
methods can be applied regardless of the internal structure of the policy (black-box optimization).
One major obstacle for search-based optimization is stochastic nature of the objective function
R(θ), which makes this problem an instance of Stochastic Optimization [12]. A possible solution
to overcome the high variance of R(θ), is to rely on a very large number of samples [10], which
nonetheless comes with high computational costs.
Our approach is based upon a Random Search (RS) [8] optimization, combined with an Objective
Estimation (OE) as objective function f : θ 7→ R+ . The OE exploits the fact that a good policy πθ ,
should perform well even in challenging conditions (worst cases). The return of one θ, is evaluated
N times, and the average of the k lowest returns is used as OE. Empirical evaluations confirmed that
such estimation scheme performs better than taking the average or the lowest value of N samples
as objective function, to optimize the TW neural circuit. To ensure that a single spuriously high
OE for some θ does not hinder the algorithm to find a legitimately good parameter, the OE must be
reevaluated after it is utilized M times, for comparisons with other OEs.

Results and Discussions
We set up the search-based RL algorithm to optimize the parameters ω, ω̂, σ, Cm , VLeak and GLeak
of each neuron and synapse of the TW neural circuit shown in Fig. (1) in the RoboSchool[11]
RoboschoolInvertedPendulum-v1 environment. The termination condition of the procedure was
set to a fixed runtime of 12 hours. The final TW neural circuit achieves the maximum possible return
of 1,000, which is equal to the performance of a PID-state-controller or an artificial neural net [11].
However, as the position and velocity of the cart are either feed into the circuit or affect the reward
directly, the controlled cart is subject to a small drift, that will make it reach one end of the movable
space after the evaluate of the environment has ended. A video demonstration of the TW controller
can be found at https://youtu.be/o-Ia5IVyff8.
In this work We showed that a model of a neural circuit from a biological organism can be adapted to
function as a controller for a task with similar characteristics to the original purpose of the circuit.
We will explore capabilities of such circuits in a future work.
4

Acknowledgments
This work was supported with computation resources by Microsoft Azure via the Microsoft Azure
for Research Award program.

References
[1] M. Chalfie, J. Sulston, J. White, E. Southgate, J. Thomson, and S. Brenner. The neural circuit
for touch sensitivity in Caenorhabditis elegans. Journal of Neuroscience, 5(4):956–964, 1985.
[2] B. L. Chen, D. H. Hall, and D. B. Chklovskii. Wiring optimization can relate neuronal structure
and function. Proceedings of the National Academy of Sciences of the United States of America,
103(12):4723–4728, 2006.
[3] C. Koch and K. Segev. Methods in Neuronal Modeling - From Ions to Networks. MIT press,
second edition, 6 1998.
[4] Z. Li, Y. Li, Y. Yi, W. Huang, S. Yang, W. Niu, L. Zhang, Z. Xu, A. Qu, Z. Wu, and T. Xu.
Dissecting a central flip-flop circuit that integrates contradictory sensory cues in C. elegans
feeding regulation. 3:776 EP –, Apr 2012. Article.
[5] R. M. Hasani, V. Beneder, M. Fuchs, D. Lung, and R. Grosu. SIM-CE: An advanced simulink
platform for studying the brain of Caenorhabditis elegans. arXiv preprint arXiv:1703.06270,
2017.
[6] A. L. A. Nichols, T. Eichler, R. Latham, and M. Zimmer. A global brain state underlies c.
elegans sleep behavior. Science, 356(6344), 2017.
[7] W. H. Press, S. A. Teukolsky, W. T. Vetterling, and B. P. Flannery. Numerical Recipes 3rd
Edition: The Art of Scientific Computing. Cambridge University Press, New York, NY, USA, 3
edition, 2007.
[8] L. A. Rastrigin. About convergence of random search method in extremal control of multiparameter systems. Avtomat. i Telemekh., 24:1467–1473, 1963.
[9] S. J. Russell and P. Norvig. Artificial Intelligence: A Modern Approach. Pearson Education, 3
edition, 2010.
[10] T. Salimans, J. Ho, X. Chen, and I. Sutskever. Evolution strategies as a scalable alternative to
reinforcement learning. 2017.
[11] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization
algorithms. CoRR, abs/1707.06347, 2017.
[12] J. C. Spall. Introduction to Stochastic Search and Optimization. John Wiley & Sons, Inc., New
York, NY, USA, 1 edition, 2003.
[13] J. G. White, E. Southgate, J. N. Thomson, and S. Brenner. The structure of the nervous system
of the nematode Caenorhabditis elegans. Philosophical Transactions of the Royal Society of
London B: Biological Sciences, 314(1165):1–340, 1986.
[14] S. Wicks and C. Rankin. Integration of mechanosensory stimuli in Caenorhabditis elegans.
Journal of Neuroscience, 15(3):2434–2444, 1995.
[15] S. R. Wicks, C. J. Roehrig, and C. H. Rankin. A dynamic network simulation of the nematode
tap withdrawal circuit: Predictions concerning synaptic function using behavioral criteria.
Journal of Neuroscience, 16(12):4017–4031, 1996.
[16] R. J. Williams. Simple statistical gradient-following algorithms for connectionist reinforcement
learning. Mach. Learn., 8(3-4):229–256, May 1992.

5

