GRSN: Gated Recurrent Spiking Neurons for POMDPs and MARL
Lang Qin Ziming Wang Runhao Jiang Rui Yan Huajin Tang


Understanding and Improving CNNs with Complex Structure Tensor: A Biometrics Study
    Josef Bigun, Fernando Alonso-Fernandez
    April 29, 2024
==================================================================================




    Spiking neural networks (SNNs) are widely applied in various fields due to their energy-efficient and fast-inference capabilities. Applying SNNs to reinforcement learning (RL) can significantly reduce the computational resource requirements for agents and improve the algorithm's performance under resource-constrained conditions. However, in current spiking reinforcement learning (SRL) algorithms, the simulation results of multiple time steps can only correspond to a single-step decision in RL. This is quite different from the real temporal dynamics in the brain and also fails to fully exploit the capacity of SNNs to process temporal data. In order to address this temporal mismatch issue and further take advantage of the inherent temporal dynamics of spiking neurons, we propose a novel temporal alignment paradigm (TAP) that leverages the single-step update of spiking neurons to accumulate historical state information in RL and introduces gated units to enhance the memory capacity of spiking neurons. Experimental results show that our method can solve partially observable Markov decision processes (POMDPs) and multi-agent cooperation problems with similar performance as recurrent neural networks (RNNs) but with about 50% power consumption.




§ INTRODUCTION

    Spiking neural networks (SNNs) possess unique spatiotemporal dynamics and all-or-none firing characteristics, enabling them to perform low-power and high-speed inference on neuromorphic hardware <cit.>. With the continuous improvement of SNN training algorithms, the learning capability and network scalability of SNNs are gradually approaching those of artificial neural networks (ANNs) <cit.>. Therefore, in recent years, there have been more and more efforts to explore SNNs as alternatives to ANNs in the field of deep reinforcement learning (DRL), aiming to reduce the computational cost and inference latency of intelligent agents.

    In SNN learning algorithms, firing rates are often used to calculate continuous values due to the discrete nature of spikes. However, the firing rates, which are in the range of 0 to 1, are difficult to map onto continuous values (such as continuous actions and Q-values) in RL that do not have value range limitations. There are generally three types of approaches to address or avoid this issue in order to achieve deep spiking reinforcement learning (SRL): (1) ANN-SNN conversion methods for RL; (2) a hybrid framework for the actor-critic (AC) method; and (3) directly trained SNNs for RL.

    
    
    The conversion algorithms <cit.> avoid the aforementioned mapping problem by using a pre-trained ANN-RL model to transform the SNN model, thus avoiding direct involvement in the learning process of RL algorithms. However, the conversion algorithms often require high simulation time steps due to the computational precision requirement, resulting in increased energy consumption and inference delay. To reduce simulation steps, a hybrid framework <cit.> for the AC method has been proposed. This approach combines spiking actor networks and artificial critic networks and utilizes a gradient descent algorithm for collaborative training. The critic networks that require precise calculations are completed by ANN, while SNN is only used for selecting actions. This hybrid framework successfully reduces the need for simulation time steps, thereby reducing energy consumption and inference delay.

    Clearly, the hybrid framework method is only applicable to RL algorithms based on the AC method. Additionally, similar to conversion algorithms, the hybrid framework method still relies on ANN to accomplish RL tasks. To enhance the generalizability of the algorithms and eliminate reliance on ANN, directly trained SNNs for RL <cit.> have been proposed. These algorithms directly train SNNs through surrogate gradient learning <cit.>, and use suitable coders to handle computations involving continuous values. These methods do not rely on ANNs and can complete tasks in a few time steps. However, such methods rely more on the design and selection of coders and also have a trade-off between performance and latency.
    
    Reinforcement learning algorithms are fundamentally designed to address sequential decision problems; they can synergize effectively with SNNs that inherently capture temporal sequence dynamics. However, the aforementioned existing SRL methods still use multiple time steps to decode the spike information to obtain continuous values, which leads to a serious temporal mismatch problem (Figure <ref>). This temporal mismatch between multiple time steps and single-step decisions not only deviates from the real dynamics in the brain but also impairs the ability of SNNs to process temporal data. 
   
    To address this issue, we propose a novel SRL paradigm that aligns the single-step updating of spiking neurons with the single-step decisions in RL, enabling the sequence decision problem to be solved within a whole simulated time window. Considering the weak temporal correlations between time steps of the original spiking neurons and the inability to guarantee accuracy after temporal alignment, we further introduced gated units to enhance the long- and short- term memory capabilities of neurons, ensuring the effectiveness of the proposed paradigm. In order to test the performance of the proposed gated recurrent spiking neurons (GRSN) and temporal alignment paradigm (TAP), we conducted experiments in partially observable (PO) and multi-agent environments. Our main contributions are summarized as follows:

    
        
  * We pointed out the issue of temporal mismatch in current SRL algorithms and proposed a novel temporal alignment paradigm (TAP) for SRL to solve this issue.
        
  * We designed a novel gated recurrent spiking neuron (GRSN) with enhanced long- and short-term memory capabilities, which can be applied to the proposed TAP.
        
  * To our best knowledge, this work is the first to use SNNs to address POMDPs and multi-agent reinforcement learning (MARL) problems. We verified the effectiveness of SNNs in PO and multi-agent environments.
        
  * Experimental results show that GRSN can outperform original spiking neurons in benchmark environments and can achieve similar performance as RNN-RL with about 50% energy consumption.
    



§ RELATED WORKS



 §.§ Spiking Reinforcement Learning
 
The development of SRL can be mainly divided into three periods: (1) the basic research of SNNs and RL <cit.>; (2) the synaptic-plasticity-based SRL algorithm <cit.>; and (3) the combination of deep RL and SNNs <cit.>. Early-stage works often study the combination of synaptic plasticity and RL theory, which aims to reveal how reward mechanisms in the brain correlate and combine with RL algorithms. In the later stage, with the development of DRL, the SRL algorithm focuses on applying SNNs to DRL, aiming for better performance.



 §.§ Recurrent Spiking Neural Networks

Due to the inherent temporal nature of spiking neurons, there have been many attempts to use recurrent spiking neural networks for temporal data processing. Some work <cit.> has designed spike-based long short-term memory networks and tested them on audio or text datasets. Other work <cit.> is to explore the long-term dependence of spiking neurons and complete structure optimization or time series forecasting tasks.



§ PRELIMINARIES



 §.§ Notations

For all variables, superscripts refer to the number of layers in the neural network, and subscripts refer to their time steps in the time window. For example, u^l_t denotes the membrane potential in layer l at the t-th time step. We follow the conventions representing vectors and matrix with bold italic letters and bold capital letters, respectively, such as o and W. We use ⊙ to signify element-wise multiplication for two vectors.





 §.§ Leaky Integrate-and-Fire Model
 
Different from traditional neural networks, SNNs use binary spikes as information carriers. In order to compensate for the lack of binary spikes in information expression, the time dimension, or latency, is introduced into SNNs. SNNs accept event streams as input, and the forward propagation of SNNs is repeated for T time steps in the temporal dimension to calculate the output spike trains. Therefore, the basic computing unit of SNNs, spiking neurons, has unique spatio-temporal dynamics. Here we introduce the widely used Leaky Integrate-and-Fire (LIF) neuron <cit.> as the benchmark in this work. In each time step t, the membrane potential u^l_t of spiking neurons at the l-th layer will integrate the input current c^l_t and the decay voltage βû^l_t-1:

    u^l_t =βû^l_t-1 + (1-β)c^l_t


    c^l_t =W^l o^l-1_t + b^l

Membrane potential will be reset to u_r (default set to 0) when a spike is emitted at the last time step:

    û^l_t   =u_ro^l_t+(1-o^l_t)u^l_t

The neurons will emit output spikes o^l_t whenever the membrane potential u^l_t exceeds the threshold ϑ:

    o^l_t   =Θ(u^l_t-ϑ)

where Θ(x) is the Heaviside function:

    Θ(x)={[ 1, if    x ≥ 0;   0, otherwise ].




 §.§ Surrogate Gradient

Due to the non-differentiability of the binary firing function Θ(x), SNNs often employ surrogate gradients <cit.> during back-propagation (BP). We adopt the arc-tangent function to replace the derivative of the binary firing function:

    Θ'(x)    ≜ h'(x) = α/2[1 + (π/2α x)^2]

where α is a hyper-parameter that controls the width of the surrogate function.
With the surrogate gradients for binary firing functions, we can directly use the BP method to train SNNs.



§ METHOD
 


 §.§ Temporal Alignment Paradigm

Reinforcement learning tasks can be modeled as Markov decision processes (MDP) (𝒮,𝒜,ℛ,p,γ), with state space 𝒮, action space 𝒜, scalar reward function ℛ, transition dynamics p, and discount factor γ <cit.>. RL agents are controlled by policy π, with the goal of maximizing the expected discounted return 𝔼_π[∑_t=0^∞γ^t r_t+1]. According to the Bellman equation <cit.>, we can obtain the iteration form of the value function:

    V^π(s_t)=𝔼_π[R_t+1+γ V^π(s_t+1)]

When the policy is determined, Eq <ref> can be simplified as:

    V(s_t)=1/γV(s_t-1) + -1/γ𝔼[R_t]

Observing Eq <ref> and Eq <ref>, we can find that the state transitions in MDPs exhibit a striking resemblance to the state changes of spiking neurons. Therefore, the key to solving the temporal mismatch problem is to utilize this similarity. 

We propose a novel SRL paradigm according to this similarity: within the framework of RL algorithms, the application of SNNs in a manner similar to RNNs (Figure <ref>A), aligning each step of spiking neuron updates u_t →u_t+1 with each state transition s_t → s_t+1 in an MDP. The advantage of this approach lies in the natural utilization of the temporal structure of SNNs while significantly reducing the number of time steps required for the entire task. Figure <ref>A and  <ref>B show the proposed TAP and the detailed dynamics of spiking neurons.



 §.§ Gradient Analysis of Spiking Neurons

In Section <ref>, we introduce the basic computational unit of SNNs: LIF neurons. It uses a constant leaky factor β to play the roles of forget gate and input gate. The use of this constant factor results in weaker temporal correlations in the neuron's internal state (u_t), making it challenging to handle data with long-term dependencies. In order to enhance the memory capacity of SNNs in the temporal domain, we need to analyze and optimize the LIF neuron. 

According to Eq <ref>-<ref>, it can be observed that the LIF neuron has two internal states: membrane potential u^l_t and input current c^l_t. To compare the importance of these two internal states in backpropagation, we need to calculate the gradient of the loss function for connection weight W^l. Suppose ℒ is the loss function that we would like to minimize. According to Eq <ref>-<ref> and the chain rule, its gradient is:

    ∂ℒ/∂W^l=∑_t=1^T ∂ℒ/∂o^l_t∂o^l_t/∂u^l_t∂u^l_t/∂W^l

T is the total number of time steps. The first term ∂ℒ/∂o^l_t in Eq <ref> is determined by the decoder and loss function and does not affect the flow of the gradient. According to Eq <ref> and Eq <ref>, the second term could be derived as:

    ∂o^l_t/∂u^l_t = ∂Θ(u^l_t-ϑ)/∂u^l_t = h'(u^l_t-ϑ)

Then we calculate the third term ∂u^l_t/∂W^l:

    ∂u^l_t/∂W^l = ∂u^l_t/∂c^l_t∂c^l_t/∂W^l + ∂u^l_t/∂u^l_t-1∂u^l_t-1/∂W^l

Bring Eq <ref> into Eq <ref>:

    ∂u^l_t/∂W^l =     [(1-β)+β∂û^l_t-1/∂c^l_t]∂c^l_t/∂W^l + 
       β∂ [u_r o^l_t-1 + (1-o^l_t-1)u^l_t-1]/∂u^l_t-1∂u^l_t-1/∂W^l

Obviously, ∂û^l_t-1/∂c^l_t = 0. Let 
    δ_t    = ∂û^l_t/∂u_t^l = ∂ [u_r o^l_t + (1-o^l_t)u^l_t]/∂u^l_t
        = [(u_r-u^l_t)h'(u^l_t-ϑ)+1-o^l_t]

Then Eq <ref> can be simplified as:

    ∂u^l_t/∂W^l = (1-β)∂c^l_t/∂W^l + βδ_t-1∂u^l_t-1/∂W^l

Continuing substitution ∂u^l_t/∂W^l leads to a closed-form expression:

    ∂u^l_t/∂W^l =    (1-β) ∑_i=1^t β^i ∏_j=1^i δ_t-j∂c^l_t-i/∂W^l + 
       ∏_k=1^t βδ_t-k + (1-β)∂c^l_t/∂W^l

When the variable subscript t=0, its value is the artificially set initial value.

To analyze the main carriers of gradient, we propose the following theorem:


With the LIF model (Eq <ref>-<ref>) and the common parameter settings (β=0.5, ϑ=1, α=2, u_r=0), the gradient of loss function ∂ℒ/∂W^l mainly depends on current ∂c/∂W^l when total time-step T is large enough.



The first term in Eq <ref> depends on the decoding method, and the second term in Eq <ref> depends on the SG function. Both of them do not affect the direction of gradient flow. Therefore, the third term (∂u^l_t/∂W^l) is the most significant. By analyzing the monotonicity (presented in the appendix) of Eq <ref>, it can be found that when T is large, ∏_i=1^T βδ_t tends to 0. Hence, the last term (1-β)∂c^l_t/∂W^l in Eq <ref> determines the direction of the gradient.


To further verify Theorem <ref>, we conducted experiments in CartPole-V (see Section <ref>) environment and visualized the parameter distribution during the training process (Figure <ref>). From figure <ref>, the maximum absolute value of the term βδ_t is less than 1, and most of the membrane potential values are distributed around 0. Hence, ∏_i=1^T βδ_t will tend to 0 when T is large enough. This result is consistent with Theorem <ref>.

According to Theorem <ref>, the gradient of the loss function ℒ is mainly transmitted through the input current c. Therefore, adding gated recurrent connections to the input current c can enhance the memory capability of spiking neurons more effectively than membrane potential u.







 §.§ Gated Recurrent Spiking Neurons

Optimizing the dynamics of neurons and improving memory ability is mainly divided into three steps. The first step is to change the decay factor β to a learnable parameter, which can be jointly optimized with the entire network. The learnable decay factor can dynamically adjust the forgetting ability of neurons during the training process and adaptively find the optimal value <cit.>. Then, we use a soft reset instead of a hard reset (Theorem <ref> still holds in the soft reset setting; see Figure <ref> and appendix for details). Compared to the hard reset method, soft reset can reduce temporal information loss, and it has better performance when the network is shallow <cit.>. Finally, we added recurrent connections and adopted gating functions to enhance the temporal correlation of the neuron's internal states c.



We named the proposed neuron model gated recurrent spiking neurons (GRSN). Its membrane potential update rules, firing rules, and definitions of Heaviside function are consistent with those of LIF neurons (see Eq <ref>, Eq <ref> and Eq <ref>). Different from LIF neurons (Eq. <ref>), the reset rule of GRSN is soft reset:

    û^l_t =u^l_t - ϑo^l_t

GRSN also adds recurrent connections at the current level:

    c^l_t = ℱ(o^l-1_t) ⊙c^l_t-1 + [1-ℱ(o^l-1_t)] ⊙ℐ(o^l-1_t)

where ℱ(·) is the forgetting gate that regulates the proportion of forget and input, and ℐ(·) is the input gate that processes input signals. Both ℱ(·) and ℐ(·) are composed of a fully connected (FC) layer and an activation function. Their detailed definitions are shown as follows:

    ℱ(x)    = σ (W_fx+b_f)
    ℐ(x)    = ReLU(W_ix+b_i)

where σ is the sigmoid function.

Figure <ref>C shows the dynamics of GRSN in the temporal alignment paradigm. Compared with Figure <ref>B, red lines indicate the recurrent connections. The pseudocode of GRSN with the temporal alignment paradigm is shown in Algorithm <ref>.



§ EXPERIMENTS
 

In the TAP, SNN behaves more like an RNN rather than a traditional DNN. Spiking neurons calculate the value function by accumulating information from multiple previous consecutive states. This configuration is often employed in RL to address POMDPs. Consequently, to validate the effectiveness and versatility of the proposed GRSN, we conducted experiments in both PO environments and multi-agent cooperative environments. The detailed parameters and settings of the experiment can be found in the appendix.



 §.§ Partially Observable Classic Control Tasks



  §.§.§ Enviroments & Experimental Settings

The Pendulum and CartPole tasks are classic control tasks for evaluating RL algorithms. The cart-pole problem described in <cit.> aims to balance the pole by applying forces in the left and right directions on the cart (Figure <ref>A). The inverted pendulum swingup problem is based on the classic problem in control theory. Its goal is to apply torque on the free end to swing it into an upright position, with its center of gravity right above the fixed point (Figure <ref>A). We conducted experiments in PO cases of these two control tasks <cit.>, in which only velocities or positions could be observed. We use suffix names (-P/-V) to distinguish these two settings.

We follow the experimental settings proposed in <cit.> and use SAC <cit.> for discrete environments and TD3 <cit.> for continuous environments. For networks, we use single-layer gated recurrent units (GRU) <cit.> for RNNs, two fully connected layers for MLP, and single-layer LIF/GRSN for SNNs. We conducted independent experiments on five different random seeds for each method and tested each final model ten times to eliminate the interference of randomness. Experiment details can be found in the appendix.









  §.§.§ Results
 
To demonstrate the superiority of the TAP and GRSN in multiple aspects, we selected three different networks and paradigm settings as baselines (Table <ref>). From the results, TAP effectively utilizes temporal information, and its performance far exceeds that of traditional algorithms using MLP. On the other hand, GRSN enhances the temporal correlation of spiking neurons, thereby further improving performance. The combination of TAP and GRSN equals or even surpasses the state-of-the-art RNN-RL method. From the learning curve (Figure <ref>), it can be seen that the convergence rate of GRSN is slightly weaker than that of traditional RNNs in some environments due to the use of discrete spikes as information carriers, but in the end, the performance of the two is very similar. LIF and MLP, which are baselines, are weaker than GRSN and RNNs in terms of temporal correlation, so their performance and convergence rate are far inferior to the other two networks.



 §.§ StarCraft Multi-Agent Challenge



  §.§.§ Enviroments & Experimental Settings

StarCraft Multi-Agent Challenge (SMAC) is a benchmark environment for evaluating MARL algorithms <cit.>. SMAC is based on the popular real-time strategy (RTS) game StarCraft 2, and it contains multiple micro StarCraft 2 scenarios (Figure <ref>B). In SMAC, the overall goal is to maximize the win rate for each battle scenario. Each agent in SMAC has a circular field of view centered on itself. With such a limited field of view, each agent is faced with a partially observable environment. The action space is discrete and includes move[direction], attack[enemy_id], stop, and no-op. Healing units use heal[agent_id] actions instead of attack[enemy_id]. The SMAC reward includes several parts: hit-point damage dealt and enemy units killed, together with a special bonus for winning the battle.

The QMIX <cit.> is a classic value-based multi-agent collaboration algorithm. QMIX adopts the centralized training distributed execution (CTDE) mode, and each agent uses RNNs as main networks. Therefore, the proposed GRSN and TAP can be well adapted to the QMIX algorithm. We follow the experimental and parameter settings of QMIX and select SMAC maps of different difficulty levels for the experiment. Detailed parameters and settings are also listed in the appendix.



  §.§.§ Results

To evaluate the performance of the method more comprehensively, we selected several SMAC maps with different difficulty levels for experiments. We conducted five independent experiments in each environment, training 10 million steps per experiment. We use the GRU-based QMIX algorithm as the baseline, and the experimental results are shown in Table <ref>. From the experimental results, it can be seen that the GRSN-based QMIX algorithm surpasses the original GRU-based QMIX algorithm in most environments, which proves that GRSN achieves or even exceeds GRU in the processing of temporal information. In addition, it is not difficult to find that the advantages of GRSN are more obvious in difficult scenarios. This also proves that GRSNs using spikes have better robustness than traditional RNNs.




 §.§ Ablation Study

In Section <ref>, we propose the TAP that can solve the temporal mismatch problem and greatly reduce time steps, and the GRSN that can enhance the temporal association of spiking neurons and improve performance. To demonstrate the impact of these two on the training speed and algorithm performance more intuitively, we conducted multiple groups of ablation experiments. We conducted experiments in both classical control tasks and SMAC environments. Similar to the previous setting, we conducted five independent experiments at each of the different settings and averaged them as the final results (Table <ref>).



  §.§.§ Temporal Alignment Paradigm

The TAP solves the mismatch problem, thus significantly reducing the number of time steps and training and inference time. From Table <ref>, the algorithm using TAP reduces the time step by T times, where T=4 is the time window size of the basic spiking neuron. In addition, TAP has also shortened the training time by about half while maintaining performance, greatly improving the efficiency of algorithms. 



  §.§.§ GRSN vs. LIF

In Table <ref>, we can compare the GRSN and LIF neurons. GRSN has better performance in different environmental and paradigm settings. However, due to the additional gating function, the training time of GRSN will be slightly longer than that of LIF. In terms of stability, GRSN also demonstrated the most stable performance, with the lowest standard deviation in five independent experiments.



 §.§ Energy Consumption Estimation

SNNs use discrete spikes for information transmission, so their energy consumption is estimated differently than that of ANNs. We follow the convention of the neuromorphic computing community <cit.> by counting the total synaptic operations (SOP) to estimate the energy consumption of SNN models. Specifically, the SOP for SNNs correlates with the neurons’ firing rate, fan-out f_out, and time window size T:

    SOP=∑_t=1^T ∑_l=1^L-1∑_j=1^N^l f_out^l[j] o_t^l[j]

where L is the total number of layers and N^l is the number of neurons in layer l, f_out is the number of outgoing connections, and o_t^l[j] denotes the output spike of the j-th neuron in layer l at the t-th time step.

For ANNs, SOP is only related to network structure. It is defined as:

    SOP=∑_l=1^L f_in^l N^l

where f_in^l represents the number of incoming connections to each neuron in layer l, L and N^l are consistent with their definitions in SNNs.

SNNs use efficient addition operations, while ANNs use more expensive multiply-accumulate operations. According to <cit.>, we measure 32-bit floating-point addition operations by 0.9pJ per operation and 32-bit floating-point multiply-accumulate operations by 4.6pJ per operation. Here, we randomly select 1024 samples to estimate the average SOP and energy consumption for SNNs. Table <ref> shows the results of different networks in different environments, and it can be seen that SNN can reduce energy consumption by up to about 3800 times compared to RNNs. The total energy consumption is also reduced by about 50%, which effectively reduces the agent's resource requirements.








§ CONCLUSION

In this paper, we introduce the temporal mismatch problem in SRL algorithms and propose the temporal alignment paradigm (TAP) and gated recurrent spiking neurons (GRSN) to solve this problem. Specifically, TAP allows the single-step update of spiking neurons to correspond to the single-step decision-making of MDP to solve the mismatch problem and significantly reduces time steps. On the other hand, GRSN enhances the temporal correlation of spiking neurons, thereby improving their ability to capture historical information and compensating for the performance degradation caused by time step reduction. Extensive experiments show that TAP can reduce the time steps by several times and half the training time, which effectively solves the temporal mismatch problem. At the same time, GRSN can also improve the performance of spiking neurons, making SNNs equal to or even exceed the traditional RNNs. Energy estimation proves that our method can reduce the energy consumption by about 50%, which provides a new algorithm option for the control task in the energy-constrained scenario.
named



§ BACKGROUND

In reinforcement learning (RL), the process of interacting with environments and changing the state through different actions is often described by Markov Decision Process (MDP) (𝒮,𝒜,p_M,r,γ), with state space 𝒮, action space 𝒜, discount factor γ, and transition dynamics p_M(s'|s,a). At every step, the agent change its state from s to s' by perform action a and receives a reward r(s,a,s'). The goal of RL is to allow the agent to maximize the expectation of accumulating rewards R_t=∑_i=t+1^∞γ^ir(s_i,a_i,s_i+1).

The agent selects actions with respect to a policy π: 𝒮→𝒜. The corresponding value function Q^π(s,a)=𝔼_π [R_t|s,a], the expected accumulate reward of agent for taking action a in state s. Rewriting the value function Q^π(s,a) into an iterative form, we can get the Bellman equation:

    Q^π(s,a)=𝔼_π [r+γ Q^π(s',a')]


Then we can define the optimal value function:

    Q^*(s,a)= max_π Q^π(s,a)


The policy π^* that achieves the optimal value function is the optimal policy. It can be obtained through greedy action choices. For high-dimensional control tasks, the value function can be fitted by deep neural networks (DNNs), which is the core idea of DRL.


§ PROOF
 
According to the proof of Theorem <ref>, the third term in Eq 9 (∂u^l_t/∂W^l) determines the direction of gradient flow. ∂u^l_t/∂W^l can be calculated as:

    ∂u^l_t/∂W^l = (1-β) ∑_i=1^t β^i ∏_j=1^i δ_t-j∂c^l_t-i/∂W^l + ∏_k=1^t βδ_t-k + (1-β)∂c^l_t/∂W^l

In Eq <ref>, both the first and third terms contain ∂c/∂W, which is consistent with the assumption that the gradient mainly flows along the current. Therefore, we only need to analyze the second term in Eq <ref>.


 §.§ Hard Reset
 
When the neuron model adopted hard reset, the δ_t can be calculated as:

    δ_t = ∂û^l_t/∂u_t^l =∂ [u_r o^l_t + (1-o^l_t)u^l_t]/∂u^l_t = [(u_r-u^l_t)h'(u^l_t-ϑ)+1-o^l_t]

so the second term in Eq <ref> can be simplified as:

    ∏_k=1^t βδ_t-k = ∏_k=1^t β [(u_r-u^l_t-k)h'(u^l_t-k-ϑ)+1-o^l_t-k]

where h'(x) is the surrogate gradient function:

    h'(x)    = α/2[1 + (π/2α x)^2]

Based on the experimental parameter settings u_r=0, ϑ=1, α=2, β = 0.5, we can simplify the multiplication term:

    β [(u_r-u^l_t)h'(u^l_t-ϑ)+1-o^l_t] 
       = 1/2 [1-o^l_t-u^l_th'(u^l_t-1)] 
       = 1/2[1-o^l_t-u^l_t/1+π^2(u^l_t-1)^2]

When u^l_t≥ϑ=1, neurons will emit spikes (o^l_t=1). Eq <ref> can be further simplified as:

    1/2[1-o^l_t-u^l_t/1+π^2(u^l_t-1)^2] 
       = -1/2u^l_t/1+π^2(u^l_t-1)^2

Let f(x)=-1/2x/1+π^2(x-1)^2, then we can analyze the monotonicity of this item.


    f(x)   =-1/2x/1+π^2(x-1)^2
    
            f'(x)   =-1/21+π^2(x-1)^2-2π^2(x-1)x/[1+π^2(x-1)^2]^2
       =-1/21+π^2-π^2x^2/[1+π^2(x-1)^2]^2

Let f'(x)=0, then:

    x   =±√(1+1/π^2)
    ⇒ max[f(x)]=f(-√(1+1/π^2))   =1/2√(1+1/π^2)/2+2π^2+2π^2√(1+1/π^2)≈ 0.0124
    ⇒ min[f(x)]=f(√(1+1/π^2))   =-1/2√(1+1/π^2)/2+2π^2-2π^2√(1+1/π^2)≈ -0.5124

According to Eq <ref>, we can find |f(x)|<1, which means |βδ_t|<1. So the term ∏_k=1^t βδ_t-k→ 0.

In another case, when u^l_t<ϑ=1, neurons will remain silent (o^l_t=0). Then Eq <ref> can be simplified as:

    1/2[1-o^l_t-u^l_t/1+π^2(u^l_t-1)^2] 
       = 1/2[1-u^l_t/1+π^2(u^l_t-1)^2]

Let g(x)=1/2(1-x/1+π^2(x-1)^2), then we can find:

    g(x) = f(x) + 1/2

Similarly, we can calculate the maximum and minimum value of g(x):

    max[g(x)]    = max[f(x)] + 1/2≈ 0.5124 
    
            min[g(x)]    = min[f(x)] + 1/2≈ -0.0124

This also satisfied |g(x)|<1, which means ∏_k=1^t βδ_t-k→ 0.

Based on Eq <ref>-<ref>, we can conclude that the second term ∏_k=1^t βδ_t-k in Eq <ref> tends to 0 when total time steps T is large. So the Eq <ref> can be simplified as:

    ∂u^l_t/∂W^l = (1-β) ∑_i=1^t β^i ∏_j=1^i δ_t-j∂c^l_t-i/∂W^l + (1-β)∂c^l_t/∂W^l

which means the gradient of loss function ∂ℒ/∂W^l mainly depends on current ∂c/∂W^l.



 §.§ Soft Reset
 
When the neuron model adopted soft reset, the δ_t should be calculated as:

    δ_t=∂û^l_t/∂u^l_t=∂ (u_t^l-ϑo_t^l)/∂u^l_t=1-ϑ h'(u^l_t-ϑ)

then the second term in Eq <ref> can be simplified as:

    ∏_k=1^t βδ_t-k = ∏_k=1^t β [1-ϑ h'(u^l_t-k-ϑ)]

where h'(x) is the same as Eq <ref>.

Similar to the hard reset mode, we bring the values (u_r=0, ϑ=1, α=2, β = 0.5) of each parameter into the equation:

    β [1-ϑ h'(u^l_t-ϑ)] = 1/2[1-1/1+π^2(u^l_t-1)^2]

Let p(x) = 1/2[1-1/1+π^2(x-1)^2], then we can analyze this item:

    ∵π^2(x-1)^2 ≥ 0 
       ⇒ 1+ π^2(x-1)^2 ≥ 1 
       ⇒ 0<1/1+π^2(x-1)^2≤ 1 
       ⇒ 0 ≤ 1-1/1+π^2(x-1)^2 < 1
       ⇒ 0 ≤ p(x) <1/2

Obviously, this also satisfied |p(x)|<1, so ∏_k=1^t βδ_t-k→ 0.



 §.§ Curves of funtions

To further confirm the correctness of the monotonicity of functions, we visualize those three functions in Figure <ref>.




§ EXPERIMENT DETAILS
 


 §.§ Experiment Settings

The experiments are all carried out on a workstation equipped with AMD Ryzen Threadripper 3970X CPU running at 2.90 GHz, 128 GB of RAM, and four NVIDIA Geforce RTX 3090 GPUs. We use Python 3.8.13 to process datasets and PyTorch 1.12.1 to implement deep SNNs. We also use the spikingjelly 0.0.0.0.14 <cit.> framework to implement deep SNN networks and custom neurons.
Table <ref> shows various hyperparameters related to spiking neurons and spiking neural networks.





 §.§ POMDP

We used the initial versions (-v0) of CartPole and Pendulum environments for testing and packaged the environment with Openai gym <cit.>. Table <ref> and Table <ref> shows all the hyperparameters in our POMDP experiments, and Figure <ref> shows the overview of network structure. The setting of each parameter and the design of the network structure all come from <cit.>.





For each set of experiments, we will save the results of the model testing during the training process, and ultimately choose the mean of the last five test results during the training process as the final result of this experiment. Then, we will select five different random seeds to independently repeat the experiment for five times, and the mean and standard deviation of the final five results will be the final result of this set of experiments.



 §.§ MARL

Due to the weak parameter settings of the original QMIX algorithm, we referred to the paper <cit.> for parameter adjustments, and the specific parameters are shown in the Table <ref>. For each independent experiment, we use the average of the last fifty testing results to represent its performance. The final results of the algorithm in different environments are the average performance of five independent experiments.





 §.§ Energy Estimation




The main text only displays the final estimation results, and here we present the complete data in Table <ref>.



