{'fraction_non_alphanumeric': 0.03870858113848768, 'fraction_numerical': 0.011180968564146134, 'mean_word_length': 4.58580106302202, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 5, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Burst-dependent plasticity and dendritic amplification support target-based learning and hierarchical imitation learning

The brain can learn to solve a wide range of tasks with high temporal and energetic efficiency. However, most biological models are composed of simple single compartment neurons and cannot achieve the state-of-art performances of artificial intelligence. We propose a multi-compartment model of pyramidal neuron, in which bursts and dendritic input segregation give the possibility to plausibly support a biological target-based learning. In target-based learning, the internal solution of a problem (a spatio temporal pattern of bursts in our case) is suggested to the network, bypassing the problems of error backpropagation and credit assignment. Finally, we show that this neuronal architecture naturally support the orchestration of hierarchical imitation learning, enabling the decomposition of challenging long-horizon decision-making tasks into simpler subtasks.


Introduction
The brain can learn a wide range of tasks very efficiently in terms of energy consumption and required evidences, motivating the search for biologically inspired learning rules for improving the efficiency of artificial intelligence. Most biologically plausible neural networks are composed so far of point neurons. Despite recent outstanding advances in this field [Nicola andClopath, 2017, Bellec et al., 2020], biologically plausible neural networks cannot achieve the state-of-art performances of artificial intelligence (e.g. they struggle to solve the credit assignment problem [Payeur et al., 2021]).
Recent findings on dendritic computational properties [Poirazi and Papoutsi, 2020] and on the complexity of pyramidal neurons dynamics [Larkum, 2013] motivated the study of multi-compartment neuron model in the development of new biologically plausible learning rules [Urbanczik and Senn, 2014, Guerguiev et al., 2017, Sacramento et al., 2018, Payeur et al., 2021.
Recent works have proposed that segregation of dendritic input (neurons receive sensory information and higher-order feedback in segregated compartments) [Guerguiev et al., 2017] and generation of high-frequency bursts of spikes [Payeur et al., 2021] would support backpropagation in biological neurons. However, these approaches require propagating errors with a fine spatio-temporal structure to all the neurons. It is not clear whether this is possible in biological networks. For this reason, in the last few years, targetbased approaches [Lee et al., 2015, DePasquale et al., 2018, Manchev and Spratling, 2020, Meulemans et al., 2020 started to gain more and more interest.
In a target-based learning framework, the targets, rather than the errors, are propagated through the network [Lee et al., 2015, Manchev andSpratling, 2020]. In this framework, it is possible to directly suggest to the network the internal solution to a task [De-Pasquale et al., 2018. However, target-based approaches require evaluating at the same time the spontaneous activity and the target activity of the network [De-Pasquale et al., 2018. This is usually solved by evaluating the two activities in two different networks, which is not natural in terms of biological plausibility.
In the present work, we show that bursts and dendritic input segregation offer a natural solution to this dilemma. In our model, pyramidal neurons rely on two different apical dendritic compartments to simultaneously evaluate the target and the spontaneous activity. A coincidence mechanism between basal and apical inputs generating the burst [Larkum, 2013] eventually defines the (target or spontaneous) spatio-temporal bursting dynamics of the network.
We exploit dendritic computation in our model, to let abstract signals act as teaching signals which drive the learning procedure in a biologically plausible fashion.
Finally, we show that this neuronal architecture naturally allows for orchestrating "hierarchical imitation learning", enabling the decomposition of challenging long-horizon decision-making tasks into simpler subtasks [Le et al., 2018, Pateria et al., 2021].

Target-based learning with bursts
We define a model of pyramidal neuron (Fig.1A, bottom) composed of three separated compartments, the basal one (i.e. the soma, receiving the sensorial input), and two apical ones, the proximal apical compartment (receiving recurrent connections from the network) and the distal apical compartment (receiving the context/teaching signal from other areas of the cortex, with a higher level of abstraction).
The spike emitted by the soma is described by variable z t i , which is equal to one when the spike is emitted at time t and zero otherwise. The spikes emitted by the proximal and distal apical compartments are described by the variables a t i and a t i , respectively. The underlying idea is that the distal compartment provides a target for the proximal one, motivating the use of the superscript symbol , which indicates the variables concerning the targets.
In addition, following [Larkum, 2013] a coincidence mechanism between the basal and the apical compartments has been implemented, yielding high-frequency bursts of spikes. In more detail, after a somatic spike, z t i = 1, a coincidence window is opened for a time interval ∆T . This is described by the variable z t i , the indicator function for t ∈ [t, t + ∆T ], which is one during this time window. If a spike is generated by the distal or proximal apical compartments within such time window, a t i = 1 or a t i = 1 with t ∈ [t k , t k + ∆T ], a high frequency burst of spikes is then produced (Fig.1B). The proximal and distal bursts can be respectively defined as i This architecture supports a burst-dependent learning rule (Fig.1A, top), enabling target-based learning. More specifically, the pattern of bursts defined by the proximal compartment (receiving the recurrent connections from the network) should mimic the ones defined by the distal compartment (which receives the teaching signal). This is possible by using the following plasticity rule for recurrent weights J b→p ij (which can be derived analytically through a likelihood maximization, see methods for details): where e t j = ∂u t i /∂J b→p ij is referred to in the literature as the spike response function [Urbanczik and Senn, 2014].
Intuitively, such plasticity rule aims at aligning in time apical proximal spikes with apical distal ones when the somatic window z t i is open. We remark that such learning rule can be computed online, and requires only observables which are locally accessible to the synapses in space and time.
As a first learning instance, we propose the storeand-recall of a 3D trajectory y t k (k = 1, . . . , 3, t = 1, . . . , T , T = 1000) in a network of N = 500 neurons (400 excitatory plus 100 inhibitory). We chose y t k as a temporal pattern composed of 3 independent continuous signals, each of which specified as the superposition of the four frequencies f ∈ {1, 2, 3, 5} Hz with uniformly extracted random amplitude A ∈ [0.5, 2.0], and phases φ ∈ [0, 2π]: A k,n cos (2πf k,n t + φ k,n ) , k = 1, 2, 3  Figure 1: Model structure A. The model of a pyramidal neuron, consisting of two separated compartments, the basal and the apical ones. The latter is further divided into two regions, proximal (receiving recurrent connections from the network) and distal (receiving teaching/context signals from other areas of the cortex).
B. In addition to isolated spike signals emitted by the soma, a coincidence mechanism between basal and apical compartments allows for the generation of high-frequency bursts of spikes. C. Store-and-recall of a 3D trajectory. The target output is automatically encoded into a spatio-temporal pattern of bursts (bottom panel), learned online thanks to the plasticity of recurrent connections, allowing for reliable reproduction of the target trajectory (top panel).
This trajectory is randomly projected through a Gaussian matrix with variance σ 2 targ to the apical (distal) dendrites of the network as a teaching signal. This input shapes the spatio-temporal pattern of spikes a ,t i from the distal apical compartment, as well as the target spatio-temporal pattern of bursts B t i (Fig.1C bottom, blue points) as described above.
A clock signal serving as a sensorial input is randomly projected (through a gaussian matrix with variance σ 2 in ) to the somatic dendrites. In more detail, the clock is here modeled as a sort of time step function with I steps, such that at each time t only component i = I · t/T is equal to one, while others are zero (see Table.1 for model parameters).
Learning is numerically implemented by several presentations of the same target trajectory y to the distal apical compartments, each time adjusting recurrent weights J b→p ij according to (1). Bursting internal activity, which represents the actual quantity mimicking the target, is translated into the output y by means of a read-out matrix J out , randomly initialized and to be trained following the rule derived by minimizing the mean squared error between the target output and the network's output: whereB t i is a time-smoothed version of burst variable B (see methods for details). At the end of the learning the plasticity of recurrent connections allows for a reliable reproduction of the target 3D trajectory (see Fig.1C, top, mse = 0.01), with an internal bursting activity reproducing the target one (Fig.1C, bottom).

Apical signals as a flexible context selection
In this section, we show that it is possible to project to the distal apical compartment signal context (through a random matrix with variance σ 2 cont ) to flexibly select and recall one of the trajectory stored in the network.
In the simplest configuration, two different context binary signals can be projected on the apical compartment, A or B ( Fig.2A). In detail, context signal is modeled as a 2D signal, which is C t = (1, 0) for the context A and C t = (0, 1) for the context B.
During the training, each context is associated with a well defined target to learn (again a 3D trajectory, as defined in the previous section, Fig.2B, left side, in red and black respectively, only one of the three trajectories is reported for simplicity). To stabilize the learning, we exploited the trick of halving the learning rates η and η out every 100 training iterations. The orthogonality of the contexts and related targets is stressed by imposing a sparsification (of 75% in the present case) in the random matrices we use to project the context and the target on the apical compartments of the network. During the recall phase, the teacher signal is no longer present, while the context signal suggests to the network which of the learned trajectory to reproduce. We show that when the context is projected to the network, the desired output is correctly recalled (Fig.2B, left side). Moreover, if the context signal is turned off in the middle of the trajectory, the network is able to self-sustain its inner dynamics, thanks to recurrent connections (Fig.2B, left side), and correctly replicate the selected trajectory.
The context is here a "suggestion", so that once started the reproduction of the correct output trajectory, the context itself becomes useless.
To demonstrate the importance to project the context signal in the apical compartments we compare these results with the case in which the context is projected in the basal ones (both during the training and the retrieval phases).
In this case, the desired trajectory is correctly retrieved when the contect is on (Fig.2B, right side).
However, we observe that the basal context is interpreted as a necessary input, so that after the turn-off the network is no longer able to sustain bursts creation, in turn causing a dramatic drop in the test performances (Fig.2B, right side). Average mean square errors, measured against both the correct target trajectory and the wrong one (i. e. the one corresponding to the other context signal), both before turn-off and after it, are provided in Fig.2D for both the neural architectures.
Furthermore, apical context architecture is also robust against corruption in the context signal, which may be the case when at higher cortex level there is only a mild preference in favor of which strategy to adopt (in comparison with the training phase, where each target is clearly and univocally associated with a sharp context signal). Here a Gaussian white noise of variance σ 2 is added during test to context signals exploited in the training (Fig.2C, top panel, σ = 0.2). The produced trajectory is similar to the trajecory targeted by the context A (black dashed line) and different from the trajectory targeted by the context B. In Fig.2C, bottom panel (blue lines) it is reported the average mse (average over 10 independent realizations of the experiment) between the output and the target trajectory (solid blue line) as a function of σ. As a reference, we also report the mse between the output and the trajectory targeted by the other context signal (dashed blue line).
It is evident a resilience of the network with apical context, while the network with basal context suddenly loses the ability to reproduce the desired output already at low levels of noise (Fig.2C, top panel and bottom panel orange lines).
At higher level of noise, basal-context network becomes in practice useless, while apical-context network is still able to reproduce the target trajectory with a remarkably small error (Fig.2C, bottom panel).

Hierarchical Imitation Learning
The proof that context can be used to flexibly choose which dynamics reproduce (and when), opens the pathway to more complicated neural architectures, naturally supporting hierarchical imitation learning. To our knowledge, no prior works are proposing biologically plausible implementations of hierarchical reinforcement or imitation learning.
We decomposed the network in two sub-networks which we call high-network and low-network (Fig.3A). The high-network (manager) computes the optimal strategy to take to solve a task and sends this information as a context signal to the low-network (worker) which actually executes it.
We applied this strategy to the so-called button & food task. In this task, an agent starts at the center of a square domain, which also features a button and an initially locked target (the food). The goal of the agent is to first press the button so to unlock the food and then reach for it. Both button and food positions are uniformly extracted in the domain [0, 1] × [0, 1]. The global task is naturally decomposed into two sub-tasks (or goals): reach button and reach food. The highnetwork computes which goal to pursue and when, and the low-network implements the sub-policy to achieve the goal.
Both the high-and the low-network share the same input, ( I = 80 input units) the vertical and horizontal differences of both the button's and food's positions with respect to agent location (∆ t = {∆x t b , ∆y t b , ∆x t f , ∆y t f } respectively). These quantities are encoded through a set of tuning curves. Each of the ∆ i values are encoded by 20 input units with different Gaussian activation functions.
To perform learning, we consider a natural hierarchical extension of behavioral cloning. The expert provides a set of hierarchical demonstrations, each consisting of low-level trajectories (to be cloned by the low-network): as well as a high-level trajectory (to be cloned by the high-network): Both state t L and state t H are the input ∆ t described above.
The action t H is the target output of the high-network and the goal t L of the low-network. It is projected to the low-network as a contextual signal in the distal apical compartment (Fig.3B, top) and is defined as a binary two-dimensional teaching signal: where χ (i) is one for vector-components i-th and zero otherwise and t b is the time when the button is reached. Intuitively, this target selects the reach button subpolicy for the first part of the task and then switches to reach target.
Given the input state t L and the context goal t L , the low-network is tasked to produce as output action t L , the velocity vector y ,t L = v t = (v t x , v t y ), where the velocities are computed so to reach the selected target in a straight line (Fig.3B, center and Fig.3C) and the output is computed as a linear readout of its internal bursting activity (Fig.3B, bottom).
The cloning procedure is implemented as a supervised learning to make the two networks reproduce the target outputs, given the input (and the context). The learning procedure is the same as the one described in Section 2.1. Finally, the two layer network is tested in closed-loop in the environment described above.
The performance in this task is measured via the following quantity:  Figure 3: Hierarchical Imitation Learning A. A two-level network, where high-level neurons produce a signal that serves as a context for the neurons in the low-level network. The two subnetworks received two different but synchronized teaching signals in the training phase. B. Button-and-food task, an agent placed at an initial position (black cross) in a 2D maze has to first, reach a button (red circle) and then the food (black circle). The high-level network chooses the order of the subtasks: reach-the-button and then reach-the-food, and projects the instruction as a contextual signal (top panel) to the apical compartments of the low-level network. The low-level network produces the output (velocities of the agent, center panel) necessary to solve the subtask as a readout of its internal bursting activity (bottom panel, blue dots). Orange dots represent the spiking activity. C. A sample spatial trajectory. Cross, red and black circles as in panel B.
where Ξ ∩ is the button-state indicator variable that is zero when the button is locked and one otherwise, the x t (·) are the agent and target position vectors and d (·, ·) is the standard euclidean distance. The condition for a successful button-press (a switch between locked and unlocked) and target-reach is taken to be d x t agent , x btn|food ≤ 0.1. Note how effectively this choice prevents the apparent divergence in the expression for ρ as the episode is stopped when the target is reached, which induces a theoretical maximum achievable score of ρ max = 10.
After the presentation of many randomly positioned button-food pairs, we observe that such twolevel network learns to correctly and efficiently solve the button & food task, with an average final score ρ = 7.3 ± 4.8 and over 90% of success rate (i.e. both button-press and target-reach conditions were met). A sample spatial trajectory produced by the network is depicted in Fig.3C.

The model
Our model of pyramidal neuron considers three different compartments: a basal one (b) and two apical ones, named proximal (p) and distal (d), respectively (see Figure 1 for reference).
Consider a particular neuron i, with i = 1, . . . , N , its real vector-valued membrane potential v t i = v t i , u t i , u ,t i (the membrane potentials of the basal, the proximal apical, and distal apical compartments respectively) follows a leaky integrate and fire dynamics, which we can generically write as: , respectively the input current, the neuron spike and the reset potential, depend on the compartment (see following sections for explicit definitions). In particular, the neural spike s t i is a stochastic variable determined via its sigmoidal probability: with v thr being the firing threshold for the membrane potential and δv a model parameter controlling the probabilistic nature of the neuron. In the δv → 0 limit, the spike-generation rule (4) becomes deterministic: We remark that we assume the deterministic limit to numerically implement the dynamics (δv → 0).

Temporal filtering and windows
We introduce the exponential filtering function filter (ξ t , τ ), defined recursively as: Basal spike signals are time-filtered through suitable time constants, depending on the direction they propagate. Using the previous definition, we introduce the following filtered quantities: Such filtering is also applied to the adaptatoin term ω t i , which is time-smoothed as: Coincidence between above-threshold somatic spikeŝ z t soma,i and apical proximal a t i or apical distal a ,t i spikes opens a time-window z t i for bursts onset The burst variable B t i ∈ {0, 1} × {0, 1} is a tuple of binary variables signaling the onset of a burst activity in the proximal or distal compartments, which can be expressed as: Aiming for a time-window variable that is active during burst activity, we can iterate the same construction developed for spikes and consider the filtered burst-onsetB One can again use this filtered quantities to introduce proximal and distal burst windows as: When at least one among proximal and distal bursts is above threshold, we finally have a neural burst activity window: which is the quantity that will feature in the dynamics of the compartments.

Basal compartment
The membrane potential of the basal compartment evolves following the equations: With J inp ik and I inp,t k respectively the input connection matrix and current, while v 0 is a compartment-specific constant input. We introduced the basal reset potential: v Where v reset,b is a compartment-specific scalar, α is a constant model parameter and B t ∨,i is the active burstwindow variable (see section temporal filtering and windows for and explicit characterization). Note how during the burst-window B t ∨,i the soma receives an extra input and the reset potential is higher, we set α = 2 and β = 20 to define the entity of such effects.

Apical proximal compartment
The apical proximal compartment of each neuron is connected to basal compartments of all the neurons through recurrent connections J b→p ij (the ones to be trained to reproduce the desired target). The equation for this compartment's dynamics are: The reset potential for the proximal apical compartment v (p) = v reset,p is a compartment-specific scalar, independent of burst activity, while u 0 is the compartment constant input.

Apical distal compartment
The signal to be learned (target) is considered as an input for the apical distal compartment: coefficient f apic is set to 1 during the learning stage, and then set to 0 to get rid of this term during spontaneous activity. Also, the input from the context (again randomly projected on the N neurons) is given as input for the apical distal compartment. The equations for the apical distal compartment read: where y ,t k is the target signal and C t k the context signal, while u 0 is the compartment constant input. We report the model parameters, for the three figures, in Table.1.

Derivation of the learning rule
We derive the update rule for the recurrent weights of the network by maximizing the probability to reproduce the target spatio-tamporal pattern of bursts, extending previous approaches used for learning target pattern of spikes [Pfister et al., 2006, Jimenez Rezende and Gerstner, 2014, Gardner and Grüning, 2016. The first step is to write the probability to produce a burst in the neuron i at time t, given the somatic window z t i . We propose the following compact formulation:  2 η and η out we report the initial parameter values, during learning they are discounted as discussed in section 2.2.
Parameter Fig 1 Fig 2 Fig 3 [L -6 (mV) u 0 -6 (mV) v thr 0 (mV) ϑsoma 2.5 × 10 −2 ϑ burst 1.25 × 10 −2 where we have introduced Φ t i (z t i ) = a t i z t i /δv −(1−z t i )γ. By definition, a burst can only happen by means of a basal-apical spike coincidence, represented by the a t i z t i term. When the basal window is open (z t i = 1) the burst probability reduces to the usual sigmoidal function. When the window is closed and z t i = 0, we have Φ t i (z t i ) = −γ, we can thus tune the γ parameter to model the burst probability. In practice, we work in the γ → ∞ limit where lim γ→∞ p(B ,t+1 i |z t i = 0) = 0, which agrees to the intuitive understanding that a closed basal window prevents any burst activity. We introduce the likelihood L of observing a given target burst activity B given the basal-to-proximal connections J b→p ij as: We can then maximize this likelihood by adjusting the synaptic connection so to achieve the target burst activity B . By differentiating with respect to the recurrent apical weights, we get: where we have introduced the following two quantities: Given the basal window z t i state, the target burst sequence is uniquely defined by the input projected to the apical distal compartment and can be written as i . If we take the model deterministic limit (δv → 0, where p(B t+1 i = 1) = a t+1 i z t i ) and note that z t i z t i = z t i , we can rewrite the previous expression in a cleaner form: This means that the spikes in the proximal apical compartment a t+1 i should mimic the ones in the distal one a ,t+1 i , when the somatic window z t i is open. For simplicity, we discussed this version of the learning rule. However, in this work we used the non-deterministic version of the rule (finite δv = 0.1) that can be rewritten as: where p a t+1 . We stress here how in the derivation we considered the basalwindows state z t i as given. Consequently, the target burst sequence B is uniquely defined by the input projected to the apical distal compartment and the likelihood is well defined. We are aware however of the feedback influence of the burst activity on the basalwindow configuration (bursts induce basal spikes, see the equation for basal current I t (b),i in the basal compartment section), we chose to neglect such contribution as it would have severely increased the difficulty of the derivation. The convergence to the chosen target thus cannot be granted. Despite the fact that we cannot theoretically prove the convergence of the learning rule, we provide a numerical demonstration that the target pattern of bursts converges to a well defined pattern (see Appendix for details).

Source code availability
The source code is available for download under CC-BY license in the https://github.com/cristianocapone/LTTB public repository.

Discussion
In the present work, we have shown that the anatomy of pyramidal neuron can naturally support targetbased learning. Moreover, it allows for using contextual signals to flexibly select the desired output from a repertoire of learned dynamics.
These properties naturally combine together to orchestrate a network with a hierarchical architecture, which in turn lends itself to hierarchical imitation learning (HIL) [Le et al., 2018]. HIL enables the decomposition of challenging long-horizon decisionmaking tasks into simpler sub-tasks, improving both learning speed and transfer learning, as skills learned by sub-modules can be re-used for different tasks. In our work, a high-level network (the manager) selects the correct policy for the task, while the low-level network (the worker) is in charge of actually executing it.
To our knowledge, there exist no other works proposing a biologically plausible architecture to implement HIL. Furthermore, our model prepares the ground for further biological explorations. Model parameters (e.g., the adaptation strength b) allows simulating the transition between different brain states (e.g., sleep and awake) [Wei et al., 2018,Goldman et al., 2020, Tort-Colet et al., 2021. Possible future investigation topics include replay of the pattern of bursts during sleep [Kaefer et al., 2020], and the effect of sleep on tasks performances [Wei et al., 2018, Capone et al., 2019.

785907) and by the INFN APE Parallel/Distributed Computing laboratory.
Appendix: Burst-dependent plasticity and dendritic amplification support target-based learning and hierarchical imitation learning A Numerical evidence of convergence As mentioned above, we can not provide a mathematical proof of the convergence toward the chosen target of burst activity by means of the learning rule proposed here. However, strong evidences in this direction can be found numerically.
We run several independent realizations of the same task of Fig.1, i.e., the store-and-recall of a 3D trajectory. We look at the distance between the target and the spontaneous spatio-temporal pattern of bursts during the training, and also at the self-distance in the pattern of spontaneous bursts across consecutive training iterations.
The parameters used for these simulations (when different from those used for Fig.1) are: η = 2.5, η out = 2.5 × 10 −3 , σ targ variable from 10 (black) to 1000 (yellow). Data averaged over 10 independent network/target realizations. The distance between two patterns of bursts A = {A t i } and B = {B t i } is defined as: For small values of σ targ , comparable to the ones used for main text figures, target bursts rapidly settle after some hundreds of training iterations (Fig.4A); within the same training scale, also spontaneous burst activity matches the target one, with a negligible error (Fig.4B). Accordingly, the overall number of bursts is the same for target and spontaneous activity (Fig.4C).
We prove that in a broad range of σ targ values, the target pattern of bursts converges to a well defined one (( Fig.4C blue dots), up to σ targ = 100 ) even though the number of bursts increases for high values of σ targ (Fig.4C red dots).