\section{Introduction} \label{Introduction}
Touch, or tactile perception, is a critical component for sensorimotor activity~\cite{romo2001touch}.
Uniquely among the senses, in many situations deliberate action by the subject is required to experience tactile feedback: this is known as \textit{active touch}~\cite{prescott2011activetouch}. Active use of touch is important for blind or visually impaired subjects, as visual perception may need to be aided or replaced by tactile (and auditory) information~\cite{Bach-y-Rita04}. One example of active touch is the braille reading, where fingers slid over lines of characters that are typeset into surfaces as a set of 1-6 embossed dots arranged in a 2$\times$3 matrix. In braille reading, characters are read sequentially, unlike print reading, where entire words or groups of words can be perceived simultaneously by the eye. However, expert users can significantly speed up their braille reading as they learn to identify various lexical, perceptual, and contextual clues~\cite{martiniello2020association}, achieving optimal reading speeds of at most 80-120 words per minute~\cite{bola2016braille}, which is about half the average silent reading rate for adults in English~\cite{brysbaert2019many}.

The sequential and time-dependent nature of braille reading makes it an excellent benchmark for machine learning applications involving time-varying signals, since braille text can be represented by sequential data acquired by moving tactile sensors. Optical classification applications for braille have been developed both in machine learning~\cite{kawabe2019experimental, li2010optical} and deep learning~\cite{hsu2020braille, shokat2020deep}. However, the image-based approach for braille recognition requires good quality for the braille samples and accurate pre-processing. Braille documents are characterized by the lack of any contrast in color between the text and the background and may require particular lighting and camera settings to produce accurate results~\cite{li2014deep}. On the other hand, automatic Braille reading relying on tactile sensors requires to add tactile sensors to robots that can slide their sensors over surfaces and the high number of sensors necessary to obtain acceptable performance determines a high overhead in terms of area, power consumption, and communication latency.

A possible solution to these problems is the inherent data encoding capabilities and sparse transmission of neuromorphic event-driven sensing~\cite{bartolozzi2016robots}. Event-driven sensors only transmit signals when a change has been detected in their sensory space, reducing communication and processing costs~\cite{bartolozzi2017event}. In fact, the event-driven domain~\cite{lichtsteiner2008128} features binary, time-discrete events, avoiding the continuous polling of sensor readouts. This is especially desirable in the domain of touch, because tactile perception is naturally sparse from a temporal and spatial perspective. Tactile events are perceived for a limited time (i.e., as long as the stimulus is applied) and in a localized part of the sensor. When no stimulus is present, the tactile system can be considered at rest.

While other event-driven neuromorphic sensors such as the \gls{dvs}~\cite{conradt2009embedded} and the silicon cochlea~\cite{chan2007aer} have attracted much interest from researchers, leading to specialized data pipelines and standardized benchmarks such as the \href{https://research.ibm.com/interactive/dvsgesture/}{DVS gesture recognition dataset} and TIDigits~\cite{leonard1993tidigits} dataset, there have been comparatively few such developments in the field of touch. Among the scarce number of examples of tactile classifiers exploiting tactile neuromorphic sensors~\cite{friedl2016human, rongala2015neuromorphic}, See et al. proposed the Spiking Tactile MNIST (ST-MNIST) dataset of handwritten digits~\cite{see2020stmnist} obtained by writing on a neuromorphic tactile sensor array. Consequently, the information in the spike patterns is mostly spatial, and a feedforward CNN reaches the best accuracy when summing up all the spikes to get a "tactile image". Bologna et al. proposed a neuroengineering framework for robotic applications, including spatio-temporal event coding, probabilistic decoding, and closed-loop motion policy adaptation for active touch. The system was benchmarked on braille, proving effective modulation of fingertip kinematics depending on character complexity. Classification of the signals recorded by a fingertip sensor mounted on a robotic arm was performed using a subset of 7 braille characters, achieving a recognition rate of (89~$\pm$~5.3)\%~\cite{bologna2013closed,pinoteau2012closed}.

In this paper, we propose a development path applicable to neuromorphic tasks in the tactile domain. The proposed method, while developed and tested on tactile output obtained from capacitive sensors, can be generalized to a whole class of time-dependent data such as audio streams, inertial sensor outputs, GPS positioning, temperature or voltage monitoring, and many others. Given the inherently time-dependent nature of its information content, we selected the braille reading problem as benchmark, for which we designed an end-to-end event-based neuromorphic classification procedure. We acquired a reliable dataset, selected encoding techniques optimized for the ratio between information content and power consumption and finally performed classification relying on a neuro-inspired approach, using \gls{snn} and \gls{rsnn} models, of which both were implemented in software and hardware. We investigated the implementations of \glspl{snn} and \glspl{rsnn} on a modular embedded GPU platform (NVIDIA Jetson) and a dedicated neuromorphic chip intended for \glspl{snn} (Intel Loihi) and benchmarked them against standard sample-based neural networks designed for time-series classification in terms of classification accuracy, average power usage, energy consumption and computation delay during inference.

We demonstrate that braille reading can be performed highly energy efficient by the use of event-based sensing and deploying spiking neural networks on dedicated hardware.

\section{Methods} \label{Methods}
\begin{figure}
    \centering
    \includegraphics[width=0.95\textwidth]{./figure/fig_01_pipeline_final}
    \caption{The workflow is summarized in five steps. Dataset acquisition and signal encoding (a) with analysis of information content and reconstruction loss (b). Different non-spiking classifiers are identified and employed to produce references for the proposed \gls{rsnn} (c), with the latter undergoing a hyperparameter optimization (d). Finally, performances are evaluated, accounting for different metrics and hardware implementations.}
    \label{fig:overall_pipeline}
\end{figure}

\subsection{The dataset} \label{methods:the_dataset}

An \href{https://www.forcedimension.com/products}{Omega.3} robot was used to slide a sensorized fingertip~\cite{jamali2015icubfingertip} over 3D printed braille letters from \textit{`A'} to \textit{`Z'} as well as \textit{`space'} at a controlled speed and position~\footnote{The robot is controlled and the sensor response is stored using YARP on a DELL XPS 15 laptop running Ubuntu 20.04 LTS.}.

\begin{figure}
    \centering
    \includegraphics[width=0.9\textwidth]{./figure/fig_02_acquisition_setup}
    \caption{The inner part of the fingertip is composed of 12 capacitance plates shown in (a), wrapped by a three layer fabric and slid over the braille letters with a constant sliding distance and velocity (b). The start position was varied following a Gaussian distribution (c).}
    \label{fig:fingertip_and_setup}
\end{figure}

The size of the braille letter was chosen to match the spatial distribution of the fingertip, so that the full letter can be detected by a single sliding movement. The sliding distance (15.5 mm), the sliding velocity (20 mm/s), and the distance to the flat surface of the plate were held constant. The start position varied following a Gaussian distribution to include spatial-temporal variability between each repetition, illustrated in Fig.~\ref{fig:fingertip_and_setup}c. Each letter was recorded 200 times with a sampling frequency of 40Hz. The capacitance value is encoded in 8-bit, leading to a range from 255-0 with a minimal change of 1, with 255 being in rest and 0 for maximum load. For convenience, the encoding was inverted in software being 0 the resting state, and 255 the maximum load of the sensor capacity.

\subsubsection{Encoding into event-based data}
A sigma-delta modulator converts the sample-based readouts from the fingertip into events. At threshold ($\vartheta$) crossings, ON or OFF digital events are generated for increase or decrease of pressure, respectively~\cite{bartolozzi2017event}.
Each original sample is converted in an offline pre-processing step from a 12 taxel time sequence to 24 event-based channels, emulating an event-based tactile sensor.

The maximum re-sampling frequency is given by the precision of the stored 64-bit float variable. Given the sensor encoding regime and the sampling frequency, the precision of the conversion is enough to encode the data with a minimum change in the sensor value of $\vartheta = 1.02\mathrm{E}-11$ without losing information from the sample-based data. A threshold value $\vartheta=1$ corresponds to the highest implemented precision and no information loss, increasing values correspond to increasing sparsity, lower data rate, improved efficiency, at the cost of lower accuracy and information loss, as shown in Fig.~\ref{fig:encode_reconstruct}.

\begin{figure}
    \centering
    \includegraphics[width=1\textwidth]{./figure/fig_03_encoding-reconstruct}
    \caption{Event-based encoding and reconstruction of a letter sample: (a) Sensor reading sequence of a sample letter along with spikes generated with asynchronous event-driven time encoding. The first sensor raw value is subtracted from the whole sequence, to replicate the change-based output. The upper part represents 250ms of the sequence of sensor readings of a single taxel sliding along the letter. The bottom part is the generated spikes for channel ON (green color) and OFF (red color) for increasing threshold values $\vartheta$, leading to decreasing numbers of events.(b) Reconstructed sequence from event-based data compared with the original sequence for a full letter sequence. Each plot represents a different value $\vartheta$ of the same sample-based sequence. Increasing the event threshold value $\vartheta$ increases the compression of the information determined by value \(\gamma\) but increases the error \(\varepsilon\) in the reconstruction.}
    \label{fig:encode_reconstruct}
\end{figure}

\subsection{Standard classifiers}\label{methods:standard_classifiers}
Non-event-driven approaches such as linear classifiers (e.g. \gls{svm}), time-series classifiers, and \glspl{lstm} were used as benchmarks on the sample-based data.

\subsubsection{Time-series classifiers} \label{method:time_series_classifiers}
We used standard time-series classifiers proven to work with time-variant datasets: \gls{fcn}~\cite{wang2017ijcnn}, \gls{resnet}~\cite{wang2017ijcnn, geng2018cscnn}, Encoder~\cite{santiago2018encoder}, \gls{tcnn}~\cite{zhao2017cnntimeseries}, and Inception~\cite{hassen2020inceptiontime}, available as \href{https://github.com/hfawaz/dl-4-tsc}{implementation on GitHub}~\cite{ismail2019deeplearning, hassen2020inceptiontime}. Please find more detailed information in the Supplementary Material.

While these networks were specifically designed for time-series, they are still based on standard feedforward structures (e.g. fully-connected layers, convolutional layers). Their lack of internal memory and recurrence requires that a whole time-series is presented to the network all at once, instead of only the data from one time step. Consequently, during inference, a buffer with the length of a time-series needs to be filled in order to perform a classification. In case of multiple overlapping time-windows, even more buffers are required. This leads to memory overhead and classification delay compared to networks with a recurrent structure.

\subsubsection{Long short-term memory} \label{methods:lstm}

Given the time-dependent characteristic of the dataset we implemented a recurrent neural network. These types of networks, in contrast to feedforward networks, incorporate an internal loop that allows temporal information to persist, and therefore are the natural choice for sequential data, but suffer from vanishing gradients for sequences characterised by long-term dependencies. This problem is avoided by using \gls{lstm} architectures~\cite{hochreiter1997lstm}, where the cell state $C_{t}$ holds long-term information. Additionally, \glspl{lstm} can add or remove information from the cell state by gates.\\
The architecture chosen for the braille dataset consists of a single layer \gls{lstm} with 234 hidden nodes, followed by a regular fully-connected layer of 234x27 output neurons that performs the classification, giving a total number of trainable parameters of 237,510. The choice was made to have the same number of trainable parameters as the best performing \gls{rsnn}, obtained through \gls{hpo} and grid search, for a fair comparison. The method for calculating the number of trainable parameters can be found in the Supplementary Material.

\subsection{Spiking neural networks} \label{methods:snn}

We designed a three-layer \gls{rsnn} to perform classification on the dataset, encoded as spike sequences with 4 different thresholds, to achieve a quantitative comparison of the different possible strategies suitable to effectively deal with time-based braille reading signals. We used the \gls{cuba} \gls{lif} neuron model written in continuous form as

\begin{equation}
\label{eq:snn1}
    \tau_{mem} \frac{\mathrm{d}U_i^{(l)}}{\mathrm{d}t} = -(U_i^{(l)} - U_{rest}) + RI_i^{(l)}
\end{equation}

with \(U_i\) being the membrane potential of neuron \(i\) (hidden state) in layer \(l\), \(U_{rest}\) being the resting potential, $\tau_{mem}$ the membrane time constant, \(R\) the input resistance, and \(I_i\) being the input current defined as

\begin{equation}
\label{eq:snn2}
    \frac{\mathrm{d}I_i}{\mathrm{d}t} = \frac{I_i(t)}{\tau_{syn}} + \sum_jW_{ij}S_j^{(0)}(t) + \sum_jV_{ij}S_j^{(1)}(t)
\end{equation}

with $\tau_{syn}$ being the synapse decay time constants, $S_j(l)$ the spike train of the $j$th neuron at the $l$th layer, $W_{ij}$ being the forward weight and $V_{ij}$ the weight of the recurrent connections.\\
Our model is based on the work by~\cite{zenke2021remarkable}.

\begin{equation}
\label{eq:snn3_a}

    I_i^{(l)}(t) = \alpha I_i^{(l)}(t-1) + \sum_j W_{ij} \cdot S_j(t)
\end{equation}

\begin{equation}
\label{eq:snn3_b}

    U_i^{(l)}(t) = (\beta U_i^{(l)}(t-1) + I_i^{(l)}(t)) \cdot  (1.0 - U_{reset})
\end{equation}

with $U_i^{(l)}$ being the membrane potential of the $i$th neuron in layer $l$, $\beta = \mathrm{exp}\frac{-time\_bin\_size}{\tau_{mem}}$ the voltage decay constant, $\alpha = \mathrm{exp}\frac{-time\_bin\_size}{\tau_{syn}}$ the current decay constant, $I_i^{(l)}$ the synaptic input current from neuron $i$ in layer $l$ multiplied by the input resistance $R = 1\Omega$ for convenience and $U_{reset}$ the reset after eliciting an event.

The error is propagated throughout the entire network, unrolled in time, using \gls{bptt}. To perform supervised learning the weight matrices $W_{ij}$ and $V_{ij}$ change following a given loss $\mathcal{L}$

\begin{equation}
\label{eq:snn4}
    W_{ij} \leftarrow W_{ij} - \eta \frac{\partial \mathcal{L}}{\partial W_{ij}} \quad \textrm{and} \quad V_{ij} \leftarrow V_{ij} - \eta \frac{\partial \mathcal{L}}{\partial V_{ij}}
\end{equation}

with learning rate \(\eta\). To use a binary step function \(\Theta(x)\) in the forward pass (inference), whose derivative is zero everywhere except the zero crossing where it becomes infinite, we use the partial derivative (the gradient) of a fast sigmoid function \(\sigma(x)\) in the backward pass (training)

\begin{equation}
\label{eq:snn5}
    \sigma(U_i^{(l)}) = \frac{U_i^{(l)}}{1 + \lambda |U_i^{(l)}|}
\end{equation}

and prevent vanishing issues. We introduce the surrogate gradient. Whereas \(\Theta(x)\) is invariant to multiplicative re-scaling, \(\sigma(x)\) needs the introduction of the scale parameter $\lambda$  being part of the hyperparameter optimization:

To compute the gradients we are using the capabilities to over-loading the derivative of spiking nonlinearity with a differential function in custom PyTorch~\cite{neftci2019surrogategradient, zenke2021remarkable}.

For the loss, we apply the cross entropy to the active read-out layer \(l = L\). For data with \(N_{batch}\) samples and \(N_{class}\) classes it is formalized as

\begin{equation}
\label{eq:snn6}
    \mathcal{L} = - \frac{1}{N_{batch}} \sum_{s=1}^{N_{batch}} \mathbbm{1}(i = y_2) \cdot \mathrm{log}\left\{\frac{\mathrm{exp}\bigg(\sum_{n=1}^T S_i^{(l)} [n]\bigg)}{\sum_{i=1}^{N_{class}} \mathrm{exp}\bigg(\sum_{n=1}^T S_i^{(l)} [n]\bigg)} \right\}
\end{equation}

whereas $n$ stands for the time step. At last we have to define the \(\mathcal{L}_1\) and \(\mathcal{L}_2\) regularization loss function.

\begin{equation}
\label{eq:snn7}
    \mathcal{L}_1 = \frac{s_l}{N_{batch}+N} \sum_{s=1}^{N_{batch}} \sum_{i=1}^{N} \bigg[\mathrm{max}\bigg\{0, \frac{1}{T} \sum_{n=1}^T S_i^{(l)} [n] - \theta_l \bigg\}\bigg]^2
\end{equation}

representing a per neuron lower threshold spike count regularization with strength \(s_l\) and threshold \(\theta_l\), and

\begin{equation}
\label{eq:snn8}
    \mathcal{L}_2 = \frac{s_u}{N_{batch}} \sum_{s=1}^{N_{batch}} \bigg[\mathrm{max}\bigg\{0, \frac{1}{N} \sum_{i=1}^N \sum_{n=1}^T S_i^{(l)} [n] - \theta_u \bigg\}\bigg]^2
\end{equation}

being a upper threshold mean population spike count regularization with strength \(s_u\) and threshold \(\theta_u\). Finally, the total loss is summarized by

\begin{equation}
\label{eq:snn9}
    \mathcal{L}_\mathrm{tot} = \mathcal{L} + \mathcal{L}_1 + \mathcal{L}_2
\end{equation}

and minimized using the Adamax optimizer~\cite{adam2017autodiff}.

To implement and simulate \glspl{rsnn} based on this model in PyTorch, we also accounted for a time binning step for the input. Although aiming to work with asynchronous and sparse event-based data, fixed sample lengths had to be defined to properly simulate algorithmic time steps in the domain of clock-driven, conventional hardware like CPUs and GPUs. Time binning was performed by subdividing the time extracted from the signal recordings ($T_{rec}$) into $T$ chunks, with $T$ defined as $T = \mathrm{int}(T_{rec}/time\_bin\_size)$ and the quantity $time\_bin\_size$ introduced as an additional hyperparameter of the \gls{hpo}. Then, by iterating over the encoded signal with a stride equal to $time\_bin\_size$, a value of 1 was assigned whenever at least one spike was found, otherwise a 0. The winning neuron in the output layer is the one with highest event count after a trial.

\subsection{Hyperparameter optimization} \label{methods:hpo}

For each event-stream produced from the original data by applying a specific threshold value, a tailored \gls{rsnn} was obtained adapting the parameter optimization procedure introduced in~\cite{fra2022human}. \gls{hpo} was performed by means of the \textit{Anneal} algorithm in the \gls{nni} toolkit, using the parameters listed in Table~\ref{tab:hpo}, over 600 trials. To partially mitigate the impact of local minima~\cite{forno2019parallel}, two evenly spaced random re-initialization of the tuner were performed during each experiment. All the trials were composed of 300 training epochs with intermediate results, for both training and test, at the end of each epoch. Test accuracy was defined as the optimization objective of the \gls{hpo} experiments, and its highest value was extracted at the end of each trial. The choice of selecting test accuracy as a reference value to be optimized was taken to account for possible overfitting. The highest training results do not always lead to the best test performances.

Following the annealing-based procedure, we performed a further exploration of the initial search space through a grid search on the two most relevant hyperparameters from the energy consumption perspective, namely $time\_bin\_size$ and $nb\_input\_copies$ since they determine the number of operations that need to be computed per inference.

As the final outcome of such a two-step \gls{hpo} problem, an optimized network for each threshold value used in the delta coding was obtained. All of these \gls{rsnn}s were composed of a recurrent, fully connected hidden layer containing 450 \gls{lif} neurons and an output layer composed of 28 \gls{lif} neurons. The number of output neurons was defined to account for an extra class, in addition to the 27 defined by the letters, suitable to identify, given future possible online implementations, edge cases such as missing contact between the fingertip and the letters. The input layer was instead part of the optimization, with its number of \gls{lif} neurons defined as $2 \cdot n\_taxels \cdot nb\_input\_copies$ where $2$ covers the event polarity, $n\_taxels$ was given by the 12 sensors in the robotic fingertip, and $nb\_input\_copies$ was to be optimized. Independently of the delta coding employed, a batch size of 128 and a learning rate $\eta = 0.0015$ were adopted for all the networks.

\begin{table}
    \caption{Description of the hyperparameters included in the search space for the HPO problem\\}
    \label{tab:hpo}
    \centering
    \begin{scriptsize}
        \begin{tabular}{>{\centering}m{0.3\linewidth}>{\centering\arraybackslash}m{0.5\linewidth}}

	        \textbf{Hyperparameter} & \textbf{Description} \\
	        \toprule
	        \textbf{scale}, $\lambda$ & Steepness of surrogate gradient \\
	        \midrule
	        \textbf{time\_bin\_size} & Time binning of the encoded input \\
	        \midrule
	        \textbf{nb\_input\_copies} & Copies of the encoded signals provided to the input layer \\
	        \midrule
	        \textbf{tau\_mem}, $\tau_{mem}$ & Decay time constant of the membrane \\
	        \midrule
	        \textbf{tau\_ratio} & Ratio between the decay time constants of the membrane and the synapse ($\tau_{syn}$) \\
	        \midrule
	        \textbf{fwd\_weight\_scale} & Scaling factor for weight initialization of the forward connections ($W_{ij}$) \\
	        \midrule
	        \textbf{weight\_scale\_factor} & Scaling factor for weight initialization of the recurrent connections ($V_{ij}$) \\
	        \midrule
	        \textbf{reg\_neurons}, $\mathcal{L}_1$ & Regularization on the number of spikes per neuron \\
	        \midrule
	        \textbf{reg\_spikes}, $\mathcal{L}_2$ & Regularization on the total number of spikes \\
	        \bottomrule
        \end{tabular}
    \end{scriptsize}
\end{table}

\subsection{Hardware implementation} \label{methods:hardware}

Beyond the algorithmic evaluation, we determined key performance metrics that are relevant to real-world deployment by implementing the networks on different hardware platforms. These metrics covered power usage, energy consumption and computation delay, and allow to draw conclusions on the deployment feasibility in real scenarios. Considering the platform related factors of high integration and availability, and our ultimate goal of deploying the algorithms in a real-life online environment on robots, we targeted the NVIDIA Jetson Xavier NX, a commercially off-the-shelf available computing platform equipped with a \gls{soc} that integrates a CPU and GPU, and the Intel Loihi, a neuromorphic processor dedicated to accelerating \glspl{snn}.

\subsubsection{NVIDIA Jetson Xavier NX}

The NVIDIA Jetson is a product family of compact and embedded computation platforms mainly targeted toward edge AI. The Xavier NX is the most powerful model among its compact 260-pin SO-DIMM modules. The form factor and connector make them easy to integrate and interchangeable. We used this general purpose platform, which is similar to common machine learning workstations in terms of architecture and software, to run all algorithms and compare the different standard time-series classification algorithms, as well as evaluate the differences between conventional algorithms and event-based algorithms on off-the-shelf hardware. Furthermore, the inference metrics give an outlook on what performance can be expected during deployment with the same hardware.

Execution time was measured by using the native functionality offered by the Linux \gls{os}. Power usage was measured by utilizing the module's onboard  \href{https://docs.nvidia.com/jetson/l4t/Tegra\%20Linux\%20Driver\%20Package\%20Development\%20Guide/power_management_jetson_xavier.html\#wwpID0E0DG0HA}{INA3221 power monitor}, polling the system's main power rail with a fixed interval of 50 ms.

The general procedure for measuring the performance metrics was performed as follows: Initially, the whole dataset was loaded into memory, followed by loading the model and its trained weights. Next, a warmup of the system was performed by letting the model predict the whole dataset in batches of 64 for six times. Its purpose was to fill the caches and to avoid additional library load times during the actual inference. Afterward, the recording of the execution time and power values were started, which was immediately followed by the inference. Like in the warmup, the whole dataset was predicted six times, but with the difference that a batch size of 1 was used to simulate how the system would behave in a real-world scenario where single samples are predicted consecutively. For our \glspl{snn} and \glspl{rsnn}, the number of samples during warmup and inference was reduced to 750 each due to timing constraints. Finally, after the inference was done, the recordings were stopped and the following metrics were evaluated:

\begin{itemize}
    \item Inference time (i.e. computation delay) per sample, which is the total inference time divided by the number of samples processed.
    \item Minimum, Maximum, and Average power usage over total inference time as well as per sample.
    \item Total and per sample energy consumption. The total energy is calculated by multiplying and accumulating each power measurement with the polling interval of 50 ms. Energy per sample is given by dividing the total energy by the number of samples processed.
\end{itemize}

Furthermore, to get more significant results, the above procedure was repeated three times per network to eliminate possible outliers and also performed for each available power mode on the Jetson.

\subsubsection{Intel Loihi}
Intel's Loihi \cite{davies2018loihi} is a fully digital neuromorphic research processor. Each Loihi chip hosts 128 neuron cores, where every neural core can run up to 1,024 \gls{cuba} \gls{lif} neurons by time-multiplexing.
The Loihi neuron's equations for the current and voltage compartments are shown in equations \ref{eq:loihi_current} and \ref{eq:loihi_voltage}, respectively:

\begin{equation}
\label{eq:loihi_current}
    I_i(t) = I_i(t-1) \cdot (2^{12} - \delta_i^I) \cdot 2^{-12} + 2^6 \cdot \sum_j w_{ij} \cdot s_j(t)
\end{equation}

\begin{equation}
\label{eq:loihi_voltage}
    U_i(t) = U_i(t-1) \cdot (2^{12} - \delta_i^U) \cdot 2^{-12} + I_i(t)
\end{equation}

where $t$ is the algorithmic time step, $I_i(t)$ and $U_i(t)$ are the current and voltage of neuron $i$, $\delta_i^I$ and $\delta_i^U$ are the current and voltage decay constants, $w_{ij}$ is the synaptic weight from neuron $j$ to $i$ and $s_j(t)$ is the spike state ($0$ or $1$) of neuron $j$.

Each Loihi neuron core supports arbitrary connection topologies as long as the capacities of the in-core memories for storing axons and synapses are not exceeded. The neuron cores are parallel and distributed with local on-chip SRAMs to store the network state and configurations. The neuron cores are fully asynchronous, performing synpatic accumulation only when there is an input event, which highly benefits from the spatio-temporal sparsity of event-based sensors and encoding. The algorithmic time-step in the entire Loihi system is maintained by a distributed handshaking mechanism called \emph{barrier synchronization}. In addition, each Loihi chip has 3 synchronous embedded x86 cores also taking part in the barrier synchronization. The x86 cores run C code and are used to monitor and interact with the SNN running on the neuron cores, handling data IO between the on-chip asynchronous neuron cores and off-chip devices, and optionally synchronizing the algorithmic time-steps duration (in physical time) to integrate the chip with a sensor.

We developed a solution to deploy the trained networks from our PyTorch implementation to Loihi (PyTorch2Loihi). First, we export the neurons hyper-parameters and the trained synaptic weights from PyTorch in a HDF5 file by taking into account the Loihi hardware specifications and constraints as follows:

\begin{itemize}
    \item {\bf Current and voltage decays constants:} We calculate the Loihi decay constants $\delta^I$ and $\delta^U$ from the PyTorch time constants $\tau^I$ and $\tau^U$ by taking into account the Loihi \gls{cuba} \gls{lif} neuron equations described in equations \ref{eq:loihi_current} and \ref{eq:loihi_voltage} as described in equation \ref{eq:loihi_decay}:

    \begin{equation}
    \label{eq:loihi_decay}
        \delta = \mathrm{int} \left( 2^{-12} - 2^{-12} \cdot e^{\frac{-time\_bin\_size}{\tau}} \right)
    \end{equation}

    \item {\bf Synaptic weights and neuron threshold:} Loihi supports up to 8-bit fixed point weights. In order to minimize the effect of quantization, we quantize the weights from PyTorch training into 256 states and adjust the weight scaling factor and threshold scaling factor to have the same overall effect of an input spike. The weight scaling factor $w_{scale}$ and the weight quantization scheme is described in equations \ref{eq:loihi_scale} and \ref{eq:loihi_weight}.

    \begin{equation}
    \label{eq:loihi_scale}
        w_{scale} = \mathrm{int} \left({\textstyle\frac{256}{\max|w|}} \right)
    \end{equation}

    \begin{equation}
    \label{eq:loihi_weight}
        w_{Loihi} = \mathrm{quantize}\left(w, \,\mathrm{step} = 2 \right) \cdot w_{scale}
    \end{equation}

    \begin{equation}
        \label{eq:loihi_threshold}
        \theta_{Loihi} = 2^6 \cdot \theta \cdot w_{scale}
    \end{equation}

\end{itemize}

After the network is deployed, we run the inference with the event-based tactile data and first quantify the classification accuracy, then, measure the energy consumption and computation delay. The test inference was made by injecting the input events of all samples of the test set in a continuous flow, where samples are separated by a blank time of $100$ algorithmic time steps where the neurons currents and voltages decay to zero.
This procedure simulates the robot sliding its finger over 3D printed braille letters. The output spikes are gathered throughout the duration of the inference, then the classification accuracy is calculated off-line.

Loihi system boards include voltage regulators and power telemetry which can be used to measure the total power consumption of the Loihi chips while a model is running. The power measurements can be combined with  timing information recorded by the on-chip x86 cores during model operation to estimate the energy consumption. NxSDK exposes high-level user interface for measuring power, energy, and timing when a workload is running. We used the interface to benchmark the performance of our \gls{rsnn} model on Loihi.

\section{Results} \label{Results}
\subsection{Standard classifiers} \label{results:standard_classifier}

\subsubsection{SVM} \label{results:svm}

Applying \gls{svm} with a linear kernel as the first approach uncovered the dataset complexity. The recorded dataset encodes information in spatio-temporal patterns. To train and test the linear \gls{svm} we run an one-vs-rest multiclass classification on the raw and the time collapsed data for 5 cross-validated splits. For the sampled-based data we achieved (87.7$\pm$1)\% and for the time collapsed (72.3$\pm$1)\% accuracy. To further investigate the temporal nature of the data we iteratively increased the number of samples taken into account from the first to the total number of samples in the sample-based stream, in every iteration the data has been reduced to the first 12 principle components found by \gls{pca} to always consider the same dimensionality of the predictors in the classifier.

In Fig.~\ref{fig:pca_svm}b the results of this procedure are shown with a clear increase in the accuracy with three significant phases at 0.5s and 0.8s before it finally saturates after 1s at 87.7\%. We can identify the three phases in the sliding procedure, namely: the first contact with the dot pattern between 0.1 and 0.5s, the first contact with the second row of the pattern between 0.5 and 0.8s, and the end of the pattern after 1s, by comparing Fig.~\ref{fig:pca_svm}b and Fig.~\ref{fig:fingertip_and_setup}c.

Similarly, for the event-based dataset we investigated the discrimination power of the spatial components removing the temporal dimension of the dataset. In this case, we built the classifier with predictors as the summation of all the events for each taxel and channel (ON/OFF) resulting in 48 predictors per letter in the dataset. Fig.~\ref{fig:pca_svm}a shows the first two principal components after applying \gls{pca} to the spatial predictors displaying an overlapping in data categories. The result of the trained classifier drops from (58.94$\pm$1.15)\% for the event-based data with all time bins as predictors to (47.7$\pm$1.48)\% for the predictors that only accounts for the sum of spatial information. This first analysis confirms the intuition that the time information of the signal carries information about the discrimination tasks even training linear classifiers.

\begin{figure}
    \centering
    \includegraphics[width=1\textwidth]{./figure/fig_04_pca_svm_plot}
    \caption{Dimensionality reduction (\gls{pca}) with 2 components applied to the spatio-temporal sequences of the sample-based dataset. Each of the colours in the visualization represents a category (letter) in the dataset. Dependency of \gls{svm} performance in regard to the first 12 principle components extracted using \gls{pca} provided with an increasing number of samples. An increase in the number of samples results in an increase in the performance close to saturation in 1s at $\sim86\pm0.77\%$.}
    \label{fig:pca_svm}
\end{figure}

\subsubsection{Time-series classifier and LSTM} \label{results:time_series_calssifier_and_lstm}

We trained the time-series classifiers and the \gls{lstm} network for 200 epochs each and averaged the results over 3 runs per network. Fig.~\ref{fig:accuracies_params_sota} shows the resulting test accuracy as well as the network's respective number of trainable parameters.

\begin{figure}
    \centering
    \includegraphics[width=0.9\textwidth]{./figure/fig_05_accuracies_parameters_sota}
    \caption{Test accuracy and number of trainable parameters of standard classifiers after training for 200 epochs and averaging over 3 runs. The networks are sorted by test accuracy in descending order.}
    \label{fig:accuracies_params_sota}
\end{figure}

\gls{resnet}, Inception and \gls{fcn} performed at comparable level. They all achieved $100\%$ in training accuracy and reached comparable test accuracy of $(98.2\pm0.24)\%$, $(97.8\pm0.23)\%$ and $(97.7\pm0.25)\%$, respectively. Their number of parameters, containing $516k$, $452k$ and $280k$, are of the same order of magnitude.

The Encoder architecture reached a training accuracy of $(99.8\pm0.1)\%$ but the drop in test accuracy was greater, resulting in $(93.9\pm0.24)\%$. This deviation is an indication for overfitting and can possibly be explained by the fact that the network's number of parameters is one order of magnitude higher than the one of the former mentioned networks. This high amount of parameters can lead to a training outcome where the network memorizes the training data and conversely generalizes worse than it would with fewer parameters.

\gls{lstm} is the only recurrent architecture in our selection. It performs sequential processing from a streaming input without buffering the data. This reduces the delay and memory footprint which is important for embedded systems. Compared to the previous networks, it achieved a lower training accuracy of $(93.7\pm1.69)\%$. Test accuracy, however, did not drop as significantly and resulted in $(90.8\pm1.29)\%$. These numbers suggest less overfitting than the Encoder network and possibly better accuracy upon further training.

\gls{tcnn} performed worst among all selected networks and only achieved $(88.5\pm0.76)\%$ and $(83.4\pm1.29)\%$ for training and test accuracy, respectively. However, it uses a greatly reduced number of parameters, containing only $2673$, which is two orders of magnitude lower than \gls{resnet}, Inception, \gls{fcn} and \gls{lstm}, and even three orders of magnitude lower than the Encoder network.

\subsection{Encoding analysis}\label{results:encoding_analysis}

To characterize the event-based datasets, we reconstructed the original time-based sequences out of the event streams and compared the results with the original sample-based signal. For every event in the event stream, the signal was increased or decreased according to the polarity, ON or OFF, by the amount defined by the threshold used in the encoding. Fig.~\ref{fig:encode_reconstruct}b shows the reconstruction values for one sample and taxel at different threshold values $\vartheta$. In Tab.~\ref{tab:encoding_analysis} we report the characterisation of the datasets properties according to the threshold value $\vartheta$, the compression ratio $\gamma$, and the reconstruction error $\varepsilon$. The compression ratio is defined as the number of events at $\vartheta = 1$ divided by the number of events at each $\vartheta$ value. The reconstruction error $\varepsilon$ is the \gls{mse} between the original sequence and the reconstructed sample-based sequence for each of the event-based datasets.

The analysis of the reconstructed sample-based signal revealed, that with increasing thresholds $\vartheta$ the number of events decreases dramatically, whereas the reconstruction error increases as expected. However, the compression rate increases with a higher slope than the \gls{mse}, showing a sparsity gain of the event-based dataset at the cost of information content.

While the number of events per taxel and letter seems to be very low, we need to consider that from the original 12 taxels not every taxel is active due to static sensor readings.

\begin{table}
    \caption{Characterisation of event-based encoding for each of the generated datasets at different threshold levels $\vartheta$. Values of mean squared error(\(\varepsilon\)) and mean events are calculated per taxel and per sample\\}
    \label{tab:encoding_analysis}
    \centering
    \begin{scriptsize}
        \begin{tabular}{>{\centering}m{0.1\linewidth}>{\centering}m{0.13\linewidth}>{\centering}m{0.13\linewidth}>{\centering\arraybackslash}m{0.13\linewidth}}
	        \multirow{2.4}{*}{} & \multicolumn{3}{c}{} \\
	        \textbf{Dataset} & $\overline{\mathrm{\textbf{Events}}}$ & $\overline{\mathrm{\textbf{Comp. ratio}}} (\gamma)$ & $\overline{\mathrm{\textbf{MSE}}} (\varepsilon)$\\
	        \toprule
	        \textbf{thr. 1} & 87.6 & 1 & 0 \\
	        \midrule
	        \textbf{thr. 2} & 38.0 & 2.3 &  0.35 \\
	        \midrule
	        \textbf{thr. 5} & 10.5 & 8.3 &  3.70 \\
	        \midrule
	        \textbf{thr. 10} & 3.4 & 25.7 & 12.29 \\
	        \bottomrule
        \end{tabular}
    \end{scriptsize}
\end{table}

Running \glspl{snn} and \glspl{rsnn} in PyTorch requires the introduction of time binning. To quantify the impact of the time binning on the different encoding datasets, we counted the total number of events given in each dataset after the time binning. The total number of events for a given encoding threshold $\vartheta$ is always higher than the total number of events for a given lower encoding threshold, regardless of the time binning, as shown in Fig.~\ref{fig:encoding_analysis}a. The higher the encoding threshold, the lower the impact of the time binning. For the encoding threshold $\vartheta = 1$, the total number of events counted in the dataset for a time bin size equal to 1 is halved with increasing time bin size to 10. For the encoding threshold $\vartheta = 2$, we still have a drop of ~34\%, whereas for higher encoding thresholds $\vartheta \geq 5$ the impact of the time binning decreases close to or below 10\% and most of the events are perceived, as shown in Fig.~\ref{fig:encoding_analysis}b. Further comparisons are reported in the Supplementary Material.

\begin{figure}
    \centering
    \includegraphics[width=1.0\textwidth]{./figure/fig_06_encoding_time_binning_analysis}
    \caption{Spike encoding: (a) Total number of events counted in the whole dataset in dependence on the selected threshold and time bin size. (b) Relative amount of events found in the dataset in relation to time bin size 1. Increasing the encoding threshold reduces the number of events significantly, whereas encoding with lower thresholds are much more affected by increasing time bin sizes. The amount of events lost is 50.37\% for $\vartheta = 1$, 34.25\% for $\vartheta = 2$, 13.17\% for $\vartheta = 5$, and only 2.92\% for $\vartheta = 10$.}
    \label{fig:encoding_analysis}
\end{figure}

\subsection{Spiking network} \label{results:snn}

The main objective for the optimization was the classification accuracy, but we also monitored the \gls{ttc} and the power consumption, with the latter separately reported in Section~\ref{results:hardware}. In the perspective of an online implementation of the proposed \gls{rsnn} an informative figure of merit to be accounted for is the minimum temporal length of the input needed for successful classification. To this aim, we defined \gls{ttc}, as the portion of the signal needed fror succesfull classification with respect to the full acquisition time of the Braille letter, that was fixed to 1.35~s, due to the fixed sliding speed.

\begin{table}
    \caption{Optimized values of the hyperparameters, for each encoding scheme, following grid search.\\}
    \label{tab:hpo_results}
    \centering
    \begin{scriptsize}
        \begin{tabular}{>{\centering}m{0.18\linewidth}>{\centering}m{0.09\linewidth}>{\centering}m{0.09\linewidth}>{\centering}m{0.09\linewidth}>{\centering\arraybackslash}m{0.09\linewidth}}

	        \multirow{2.4}{*}{} & \multicolumn{4}{c}{\textbf{Delta coding threshold}} \\
	        \cmidrule(lr){2-5}
	        & \textbf{1} & \textbf{2} & \textbf{5} & \textbf{10} \\
	        \toprule
	        \textbf{scale} & 5 & 15 & 10 & 10 \\
	        \midrule
	        \textbf{time\_bin\_size} \tiny{(ms)} & 5 & 3 & 3 & 5 \\
	        \midrule
	        \textbf{nb\_input\_copies} & 2 & 8 & 4 & 2 \\
	        \midrule
	        \textbf{tau\_mem} \tiny{(ms)} & 60 & 50 & 70 & 70 \\
	        \midrule
	        \textbf{tau\_ratio} & 10 & 10 & 10 & 10 \\
	        \midrule
	        \textbf{fwd\_weight\_scale} & 1 & 1 & 1.5 & 4 \\
	        \midrule
	        \textbf{weight\_scale\_factor} & 1e-2 & 2e-2 & 3.5e-2 & 1.5e-2 \\
	        \midrule
	        \textbf{reg\_spikes} & 4e-3 & 1.5e-3 & 1e-3 & 1.5e-3 \\
	        \midrule
	        \textbf{reg\_neurons} & 1e-6 & 0 & 0 & 0 \\
	        \bottomrule
        \end{tabular}

    \end{scriptsize}
\end{table}

The parameter space after the \gls{nni} optimization, as well as after the grid search, for each threshold shows no significant trends. The complex interaction of different parameters leads to a variety of local optima resulting in comparable test accuracy. Looking only at the trials with the best classification accuracy for different encoding thresholds, reported in Tab.~\ref{tab:hpo_results}, gives the same picture. Only the forward weight scale, $fwd\_weight\_scale$, seems to be constantly increasing with increasing thresholds. The membrane potential time constant $\tau_{mem}$ has only slight variations, again with no clear trend, and the $tau\_ratio$, describing the relation of $\tau_{mem}$ and $\tau_{syn}$, is constant. Seeing similar membrane time constants indicates their dependencies on the spatial-temporal properties of the data despite the encoding threshold. A constant synapse time constant, in contrast, shows to be an optimal relation between neuron and synapse dynamics for this task.

Fig.~\ref{fig:acc_plot} shows a summary of the \gls{rsnn} classification accuracy during grid search optimization. From all the explored combinations of $time\_bin\_size$ and $nb\_input\_copies$ for all the encoding thresholds $\vartheta$ employed, the best configuration in terms of accuracy turned out to be the one adopting an encoding threshold of 5, a time bin size of 3 ms and 4 input copies. Nevertheless, such \gls{rsnn} configuration did not provide the strongest reliability in terms of repeatability. As shown in Fig.~\ref{fig:acc_plot}b, the standard deviation of the provided results is larger than the one observed for other configurations. Fig.~\ref{fig:acc_plot}b shows that the best configuration found for an encoding threshold of 2 resulted in a mean accuracy as high as the one for an encoding threshold of 5 but with a significantly reduced standard deviation: (80.9~$\pm$~0.3)\% test accuracy compared to (80.9~$\pm$~1.9)\%.

Looking at the accuracy performances from this twofold perspective, it is hence possible to identify as best configuration the one employing the encoding threshold of 2 with a time bin size of 3 ms and a number of copies equal to 8. The overall performance of the highest threshold ($\vartheta = 10$) is the worst. The analysis of accuracy results provided an additional insight also, reported in both Fig.~\ref{fig:acc_plot}a and Fig.~\ref{fig:acc_plot}b, highlighting that the encoding threshold and $time\_bin\_size$ can have a significant impact. Particularly,  they induce a similar behavior of test accuracy: after an initial growth leading to a maximum, a further increase of one of them produces a deterioration of the classification performance. In contrast to the findings regarding the preservation of events concerning the time bin size for different thresholds discussed in \ref{results:encoding_analysis}, the accuracy for higher encoding thresholds decreases most for higher time bin sizes. Comparing the network performance of the \gls{rsnn} and the feedforward \gls{snn}, shown in Fig.~\ref{fig:acc_plot}b, we observe a constant decrease for the feedforward \gls{snn} with increasing thresholds, but a constant performance for the \gls{rsnn} up to $\vartheta = 10$, where it initially starts to drop off.

Targeting an online hardware implementation, next to classification performance, the energy efficiency needs to be considered too. From that point of view, the encoding threshold of 1 is the most promising candidate using a small number of input copies and a greater time bin size, leading to a significant lower energy footprint at comparable performance.

Regardless of the hyperparameters, the \gls{ttc} is equal throughout all conditions and the whole time series is needed for the best classification performance, similar to the findings using the linear \gls{svm}, reported in Sec.~\ref{results:svm}.

\begin{figure}
    \centering
    \includegraphics[width=1.0\textwidth]{./figure/fig_07_parameter_acc_plot}
    \caption{Accuracy of \gls{rsnn} and \gls{snn}: (a) Best test accuracy results achieved with the \gls{rsnn} for all the combinations of $time\_bin\_size$ and  $nb\_input\_copies$ explored with grid search for each encoding threshold. (b) Comparison of feedforward \gls{snn} and the \gls{rsnn} for the best combination of $time\_bin\_size$ and $nb\_input\_copies$ for each encoding threshold.}
    \label{fig:acc_plot}
\end{figure}

\subsection{Hardware implementation} \label{results:hardware}

\subsubsection{NVIDIA Jetson embedded GPU} \label{results:hardware_jetson}

Fig.~\ref{fig:power_state_of_the_art} shows the energy consumption, average power usage and inference time of the standard classifiers running on the NVIDIA Jetson. We consider energy consumption the main metric as it includes both power usage and inference time per sample. Average power usage gives insight on what power budget would be required to achieve certain inference times. Both, energy and power, are important for battery powered applications. Finally, inference time is crucial in applications with real-time constraints.

\begin{figure}
    \centering
    \includegraphics[width=0.9\textwidth]{./figure/fig_08_power_evaluation_sota}
    \caption{Comparison of inference metrics for the standard classifiers in terms of energy consumption and average power usage as measured on an NVIDIA Jetson Xavier NX. The networks are sorted by their corresponding test accuracy in descending order. The label on top of each bar shows the inference time per sample on the corresponding network.}
    \label{fig:power_state_of_the_art}
\end{figure}

When comparing these results with the parameter counts in Fig.~\ref{fig:accuracies_params_sota}, some similarities can be found. \gls{resnet}, Inception and \gls{fcn}, which are comparable in terms of accuracy and parameter count, had nearly the same average power usage during inference. This means that their energy consumption is directly proportional to their inference time. In case of Inception and its parameter count, we expected the energy consumption to lie between \gls{resnet} and \gls{fcn}, but it actually exceeded both. While parameter count is not directly related to computational complexity, another explanation for this observation could be that Inception uses operations that are less optimized or not accelerated by the GPU. The similarities continue with the Encoder, where average power usage jumped and the parameter count increased by one order of magnitude, compared to the former three networks. Judging only by the parameter count, we expected energy consumption to be even higher. The difference, especially compared to Inception, is not as great as expected, though. Considering the higher average power usage, it had better GPU utilization and therefore could benefit from an overall higher acceleration. Inference time also supports this assumption, as it was lower on the Encoder than on Inception.

Amongst all standard classifiers, the results for the \gls{lstm} are the most notable. Despite being a sequential network, which we expected to be slower compared to the standard classifiers due to its iterative and recurrent nature, it achieved the fastest inference time and lowest energy consumption. But this came at the cost of the highest power consumption. Similar to the Encoder, this suggests that the network can benefit from a high GPU utilization or general acceleration of internal operations.
Lastly, the \gls{tcnn} performed poorly when putting it alongside the other networks. It had, on average, a similar energy consumption but a high inference time, despite having by far the least amount of parameters of the shown networks. In general, it seems that this specific architecture is not very well suited for solving the problem at hand.

For the \glspl{snn} and \glspl{rsnn}, we expected results on the NVIDIA Jetson to be solely proportional to the parameters $time\_bin\_size$ and $nb\_input\_copies$, and whether the feedforward or recurrent architecture was used. These three factors mainly define the number of operations to be calculated during inference. Conversely, we assumed that the threshold does not affect performance as the implementation on general purpose computers does not take advantage of the temporal sparsity in the data. Our measurements generally support these assumptions and are shown in Fig.~\ref{fig:power_spiking}. The average power usage of the feedforward and recurrent architectures across their thresholds were very close, deviating less than $0.7\%$ for the former, and less than $0.2\%$ for the latter from their respective means. This implies a constant utilization of computational resources as well as the energy consumption being directly proportional to the inference time for each architecture. Thresholds 1 and 10 as well as thresholds 2 and 5 consumed roughly the same amount of energy per inference. Tab.~\ref{tab:hpo_results} shows that both threshold pairs presumably depend on $time\_bin\_size$ and $nb\_input\_copies$. The only exception is $nb\_input\_copies$ for threshold 2 and 5, which is 8 and 4, respectively. This brings us to the conclusion that $nb\_input\_copies$ does not have a meaningful impact on energy consumption and inference time in real-life scenarios, whereas $time\_bin\_size$ and the type of architecture are the main factors for computational load.

\begin{figure}
    \centering
    \includegraphics[width=0.9\textwidth]{./figure/fig_09_power_evaluation_snn}
    \caption{Comparison of inference metrics for all spiking neural networks in terms of energy consumption and average power usage as measured on an NVIDIA Jetson Xavier NX. The label on top of each bar show the inference time per sample on the corresponding network.}
    \label{fig:power_spiking}
\end{figure}

In general, when comparing the absolute numbers of Fig.~\ref{fig:power_spiking} with the standard classifiers in Fig.~\ref{fig:power_state_of_the_art}, our implementations of the \gls{snn} and \gls{rsnn} have a clear disadvantage at energy consumption and inference time when being run on a GPU accelerated device. The most efficient spiking network consumed $\sim$88\% more energy than the least efficient standard classifier, and the fastest spiking network took $16.8 \times$ longer for one inference than the slowest standard classifier. These numbers clearly show the need for dedicated neuromorphic hardware to implement event-based algorithms.

\subsubsection{Intel Loihi neuromorphic chip} \label{results:hardware_loihi}

\begin{figure}
    \centering
    \includegraphics[width=0.5\textwidth]{./figure/fig_10_loihi_plot_accuracy}
    \caption{The comparison of feedforward \gls{snn} and the \gls{rsnn} accuracy on Loihi for the best combination of $time\_bin\_size$ and $nb\_input\_copies$ for each encoding threshold.}
    \label{fig:loihi_accuracy}
\end{figure}

The overall trend of accuracy in Loihi for the feedforward \gls{snn} (shown in Fig.~\ref{fig:loihi_accuracy}) follows the trend of accuracy on the PyTorch simulations (and Jetson inference) shown in Fig.~\ref{fig:acc_plot}b. Nevertheless, there is a loss in accuracy of a few percents (e.g. $-1.58\%$ for the \gls{rsnn} with encoding threshold $1$), which is due to the PyTorch training procedure where we did not take into account the Loihi hardware constraints, in particular the 8-bit fixed point weights implementation. The loss varies depending on the PyTorch weights distribution.

\begin{figure}
    \centering
    \includegraphics[width=0.9\textwidth]{./figure/fig_11_loihi_plot_energy}
    \caption{Comparison of inference metrics for all trained spiking neural networks in terms of energy consumption and average power usage as measured on Loihi. The label on top of each bar show the inference time per sample on the corresponding network.}
    \label{fig:loihi_energy}
\end{figure}

Then, we compared the hardware efficiency of the recurrent and feedforward \gls{snn} in terms of delay (i.e. execution time), power and energy consumption. Before discussing the results, it is important to specify the neural cores mapping we have used which does not affect the accuracy but does affect the hardware efficiency. In fact, Loihi offers flexibility in how to map the network neurons into the neural cores, constrained by the number of cores in a chip as well as the number of input axons, synapses, neurons and output axons in a neural core. The goal is then to find a good trade off between parallelism (i.e. using more neural cores with less neurons per core) and time-multiplexing (i.e. using less neural cores with more neurons per core), in order to balance the neural cores power, the mesh routing power and algorithmic time step duration to get an optimal configuration with respect to the application requirements. We found, that for the specific typologies, the more cores we use the more power and energy we consume without a significant impact on the delay. We therefore used the minimum number of cores considering all the hardware constraints which is $8$ cores for all the trained networks.

Fig.~\ref{fig:loihi_energy} shows the delay, power and energy consumption of the different deployed \glspl{snn} on Loihi\footnote{Energy and timing measurements were obtained on Nahuku 32 board ncl-ext-ghrd-01 with an Intel(R) Xeon(R) CPU E5-2650 0 @ 2.00GHz and 4GB RAM running Ubuntu 20.04.4 LTS and NxSDK v1.0.0}. For the sake of simplicity, we refer to the energy consumption as the main metric, as it includes both power and delay. As expected, feedforward \glspl{snn} consumes less energy than \glspl{rsnn}, for the same thresholds. This is due to the overhead of memory and computation from the recurrent synaptic connections. Feedforward \glspl{snn} and \glspl{rsnn}  follow a similar trend for the different thresholds: networks with thresholds $2$ and $5$ consume more energy because they have smaller bin sizes and therefore more time bins (i.e. algorithmic time steps) per sample ($450$) compared to networks with thresholds $1$ and $10$ ($270$), and they have more input copies as shown in Tab.~\ref{tab:hpo_results}. Networks with threshold $2$ consume more than networks with threshold $5$ mainly because they have more input copies ($8$ vs. $5$). Nevertheless, the feedforward \gls{snn} with threshold $1$ consumes more than the feedforward \gls{snn} with threshold $10$, while the \gls{rsnn} with threshold $1$ consumes less than the \gls{rsnn} with threshold $10$. Even though in both cases the networks with threshold $1$ have more events in the input and less events in the hidden layer than the networks with threshold $10$, shown in  the Supplementary Material, the impact of the hidden layer events is different, because every event in the feedforward \gls{snn} hidden layers gets transmitted to the $28$ output neurons while every spike in the \glspl{rsnn} hidden layers gets transmitted to both the $28$ output neurons and the $450$ hidden neurons. Therefore, the gain obtained in the input layer for the \gls{rsnn} with threshold $10$ is lost with the recurrent topology which increases the number of synaptic operations. Finally, while the Jetson GPU is mostly sensitive to the number of bins as shown in Fig.~\ref{fig:power_spiking}, Loihi is also sensitive to the number of input copies and the spatio-temporal sparsity of the spikes and synaptic operations in the network.

\begin{table}
    \caption{Comparison of RSNN on Loihi, RSNN on Jetson and LSTM on Jetson in accuracy, power, energy, delay and energy delay product. The number of trainable parameter (i.e. synaptic weights) is similar between the RSNN (236,700) and the LSTM (236,892). The accuracy gains are calculated as a difference while all the other gains are calculated as ratio. For all gains, positive values mean improvements for the RSNN on Loihi.}
    \label{tab:loihi_comparison}
    \centering
    \begin{scriptsize}
        \begin{tabular}{lccccc}
        \hline
        \multirow{2}{*}{\textbf{Metric}} &
            \multirow{2}{*}{\textbf{\makecell{RSNN on \\ Loihi}}} &
            \multirow{2}{*}{\textbf{\makecell{RSNN on \\ Jetson}}} &
            \multirow{2}{*}{\textbf{\makecell{LSTM on \\ Jetson}}} &
            \multicolumn{2}{c}{\textbf{RSNN on Loihi gain vs.}} \\ \cline{5-6}
                                      &       &         &       & \textbf{RSNN on Jetson} & \textbf{LSTM on Jetson} \\ \hline
        Accuracy (\%)                 & 78.32 & 79.90    & 92.56 & -1.58                  & -14.24                  \\ \hline
        Total power (mW)              & 31    & 3,851    & 7,424  & 125 $\times$             & 240 $\times$              \\ \hline
        Total energy per sample (uJ)  & 71    & 1,108,695 & 16,800 & 15,612 $\times$           & 237 $\times$              \\ \hline
        Delay per sample (ms)         & 2.30  & 295.38  & 2.30  & 129 $\times$             & 1 $\times$                \\ \hline
        Energy Delay Product (uJ * s) & 0.16  & 327,485.00  & 38.64 & 2,006,475 $\times$         & 237 $\times$              \\ \hline
        \end{tabular}
    \end{scriptsize}
\end{table}

After the comparison of the different deployed \glspl{snn} on Loihi, taking into account the accuracy, power, energy and delay, we conclude that the \gls{rsnn} with encoding threshold $1$ in the best option. For simplicity, we will refer to it as the \gls{rsnn}. Tab.~\ref{tab:loihi_comparison} compares \gls{rsnn} in Loihi, the \gls{rsnn} on Jetson and the \gls{lstm} on Jetson. The \gls{rsnn} on Loihi loses $1.58\%$ in accuracy compared to the \gls{rsnn} on Jetson. It further loses $14.24\%$ compared to the \gls{lstm} on Jetson. A specific \gls{lstm} architecture with the same input copies strategy and number of parameters as in the \gls{rsnn} has been used and trained for the same number of epochs (300), which improved the accuracy from $90.8\%$ (as shown in Fig. \ref{fig:accuracies_params_sota}) to $92.6\%$ due to the input copy. However, the \gls{rsnn} on Loihi shows several orders of magnitude gains in hardware efficiency. First, compared to the \gls{rsnn} on Jetson, it is $125$ times more power-efficient and $129$ times faster, which makes it four orders of magnitude ($15612 \times$ more energy-efficient. It clearly shows that \glspl{snn} are particularly inefficient when implemented on standard GPU hardware. It should nevertheless be highlighted that the delay or execution time of the \gls{rsnn} on Jetson still satisfies the real-time constraint imposed by the sensor which has a sampling frequency of $40$Hz (i.e. a maximum algorithmic time step duration of $25$ms). Even though the average execution time of the \gls{rsnn} on Jetson is relatively long ($295.38 ms$), it is still lower than the total duration of each sample ($1350 ms$). It is to note that this delay can increase in the real-world setting when adding off-chip communication with the robot. Second, compared to the \gls{lstm} on Jetson, the \gls{rsnn} on Loihi is $240 \times$ more power-efficient and exhibits a similar average execution time, which makes it two orders of magnitude ($237$ times) more energy-efficient.

Overall, compared to the standard time-series classifiers with the Jetson GPU and the \gls{lstm}, the neuromorphic approach with the Loihi chip and \glspl{rsnn} is about $14\%$ less accurate but $240 \times$ more power-efficient, reducing the total power from $7.424 W$ to about $31 mW$. Since the execution time is similar, there is a similar gain of $237 \times$ in energy consumption and energy-delay product. This goes in line with recent results of spiking neural networks on Loihi compared to standard algorithms and hardware, where the best performing workloads on Loihi make use of highly recurrent networks \cite{Davies_etal21}. Furthermore, we can expect an even higher gain in energy-efficiency when using the \gls{rsnn} on Loihi in the real-world environment, because it exploits the spatio-temporal sparsity of the event-driven encoding. Therefore, if the robot is not moving its finger, no event is transmitted to the Loihi chip, drastically reducing the dynamic power that represents about $20$mW out of the total $31$mW. Instead, the Jetson GPU would always process the redundant samples coming from the sensor. Our study shows how event-driven encoding, neuromorphic hardware and \glspl{snn} put together improve the overall efficiency of the tactile patterns recognition, emphasising on the importance of the neuromorphic approach for embedded applications with a continuous stream of data.

\section{Conclusions} \label{Conclusions}
The initial analysis of the sample-based data demonstrates that the information in our dataset is encoded in the temporal domain, thus motivating the use of architectures that are capable to learn spatio-temporal patterns from the data. The results show a decrease in the accuracy of linear classifiers when no time dimension is accounted for in both sample- and event-based data. Although linear classifiers provide a high accuracy when all time bins are taken into account as predictors, it is not a desired approach to learn spatio-temporal patterns since they are not able to scale with the time sequence of events. This is in contrast with the sequential learning approaches and spike-based algorithms explored in this manuscript.

The data encoding analysis presents the trade-off between information content from the original sample-based data and the sparsity of the event stream. The original sample-based data is inherently redundant since the information content decreases slower than the compression ratio of events revealed by comparing the datasets at different $\vartheta$ threshold levels.
In comparison with the sample-based dataset, an increasing encoding threshold in the event-based dataset increases the sparsity of the data. But based on the implementation results, the network may not reflect this sparsity at every network layer. An optimized network for task performance could increase the quantity of events and energy consumption of the total architecture although the dataset presented to the input layer have a compression ratio bigger than 1 in terms of events. Therefore, encoding schemes that minimize the number of events in the input may result in more energy consumption in the system as a whole. This is highlighted in our hardware implementation with an increase in energy consumption for $\vartheta$ values 2 and 5 with respect to 1 as shown in Fig.~\ref{fig:power_spiking} and Fig.~\ref{fig:loihi_energy}), and in terms of increase in the total number of events in the system as reported in the Supplementary Material.

We further found that the spiking neuron current and voltage time constants that were optimized independently for each encoding threshold are similar, because they are correlated to the inherent temporal dynamics of the input event-stream rather than the encoding threshold or binning time window. The \gls{hpo} did not settle at global optima, but underlines the high complex interaction of the included parameters, leading to many locally optimal solutions. Looking at the results of the follow-up grid search reveals, that all time bin sizes greater $2$ms lead to similar/comparable classification accuracies, with a slight decrease for increasing time bin sizes. Relating the moderate decrease of accuracy with the increasing in energy/power saving makes the selection of higher time bin sizes for further robotic implementation preferable.

Our findings regarding the implementation on the NVIDIA Jetson show that it is capable of fulfilling the constraints of real-time performance, which is defined here as the inference time of any network being lower than the recording time of one sample. The increase of inference time between non-spiking and spiking architectures is substantial, though. It ranges from $\sim10\times$ (Inception vs. \gls{snn} with threshold 1) to $\sim214\times$ (\gls{lstm} vs. \gls{rsnn} with threshold 2). As a consequence, energy consumption also rose by a great margin which ranges from $\sim1.9\times$ (Encoder vs. \gls{snn} with threshold 1) to $\sim108\times$ (\gls{lstm} vs. \gls{rsnn} with threshold 2). These numbers show the clear drawback of our implementation on conventional hardware. For a meaningful deployment it would either need a more optimized implementation that is better accelerated by GPUs, or dedicated hardware that can take advantage of the characteristics of the spiking domain, like temporal sparsity.

We have been able to demonstrate the possibility of efficiently performing time series classification by exclusively using event-based encoding and asynchronous spike-based computation on neuromorphic hardware. Our deployed \gls{rsnn} is able to discriminate between 27 classes of braille letters with $78.32$\% accuracy, using only 450 recurrently connected hidden units and consuming a total amount of $31$mW on the Intel Loihi neuromorphic chip. Comparing these findings to a \gls{lstm} on the nVidia Jetson embedded GPU yields a gain in power-efficiency of $240\times$ for the \gls{rsnn} on Loihi. That makes it especially attractive for mobile robots or highly energy constrained fields of applications. In addition to the neuromorphic computation while performing the task, the event-based encoding is important in this context. Event-based systems can be considered at rest while no significant change occurs and with that, the power consumption during that time is extremely low. Nevertheless, the system is ready to work at any time, due to its asynchronous nature. In contrast to this offline benchmarking methodology, an online classification can be performed by implementing the output layer as a winner-take-all network, to be able to read out the network prediction at any point in time. Another possible option is to include the inter-spike-interval (ISI) in the optimization framework and constantly monitor it during inference, leading again to the possibility to select the winning neuron at each point in time.

Nevertheless, the \gls{lstm} outperforms the \gls{rsnn} in accuracy by $14.21$\%. Several mechanisms can be explored to improve the pattern recognition performance of the \gls{rsnn}. First, by only applying a single recurrent hidden layer, we limited the trainable parameters in our network and with that, its learning capabilities. Future investigations might use multiple hidden layers with reduced number of hidden units to increase the spatio-temporal encoding power. Second, at the neuron level, we used homogeneous time constants for all neurons. Learning the time constants along with the synaptic weights to adapt the different neurons temporal dynamics has been shown to be beneficial for the classification performance~\cite{Perez-Nieves_etal21}. Third, recent works suggest the need for more powerful recurrent units for spiking neurons in order to bridge the gap with \gls{lstm} and other formal (i.e. non-spiking) recurrent neuron models \cite{He_etal20,ParedesValles_etal21}. Despite the energy benefits of neuromorphic hardware, neuromorphic algorithmic space is still being explored, therefore, not as mature as standard ANN methods. We envision that this problem and the performance metrics we used will serve as a benchmark to drive progress in the field.

The fingertip sensor used for the dataset acquisition has a very limited spatial resolution, nonetheless sufficient for the task according to the accuracy of the classification outcome. Upcoming investigations with high-resolution tactile sensors will open the door for more complex tasks, fully benefiting from the rich spatial-temporal nature of the event stream, extracting spatial and temporal pattern simultaneously. Also the deployment of models mimicking the biological skin with its wide range of neuronal responses, e.g. slow and fast adaptive to name only two, can be beneficial in terms of information extracted from the stimuli. By looking at humans performing braille reading, we can see the strategy of multi-finger usage as an improved scanning strategy. The use of multiple fingers, or sensors in the artificial domain, in parallel, one next to the other, could improve the performance by providing redundant information and with that reducing the impact of noisy sensor readings.

Overall, we presented a tactile spatio-temporal dataset suitable for benchmarking event-based encoding schemes and neuromorphic algorithms. We compared an event-based encoding and spike-based learning algorithm with standard machine learning approaches and implemented a classification system into different hardware platforms showing the advantages of the end-to-end neuromorphic approach for tactile patterns recognition. Nevertheless, our findings are not limited to tactile data, because spatio-temporal information is present in all sensory modalities when processed in a streaming fashion at the edge, and our approach can be applied to these types of workloads. Building full neuromorphic systems that inherently sense, process and communicate their output in an event-based manner provide great potential in terms of energy efficiency and scalability for embedded and \textit{embodied} sensory-motor systems that interact in real-time with the real-world environment.

\section*{Conflict of Interest Statement} \label{Conflict of Interest Statement}
The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

\section*{Author Contribution} \label{Author Contribution}
SMC had the initial idea and recorded the dataset. APZ and SMC performed the sample-to-event conversion. VF, EF and DK performed the NNI implementation. VF, DK, APZ, SMC run the optimization. VF and SMC implemented the performance evaluation. APZ, DK, SMC and DI investigated standard classifiers and carried out encoding analysis. APZ, SMC and SR investigated linear SVM and obtained PCA plots. DK acquired GPU measurements. LK deployed the \glspl{snn} on Loihi and acquired the measurements. LK, GU, FZ, and CB supervised the work. All authors contributed to the manuscript writing and approved the submitted version.

\section*{Acknowledgement} \label{Acknowledgement}
The authors would like to acknowledge the 2021 Telluride neuromorphic workshop and all its participants for the fruitful discussions, and Intel Corporation for access to the Loihi neuromorphic platform. The authors would like to thank Sumit Bam Shrestha from Intel Neuromorphic Computing Lab for his support on the Loihi implementation and feedback on the manuscript. The research activity herein was partially carried out using the HPC infrastructure as well as the NVIDIA Jetson platform of the Silicon Austria Labs (SAL), the HPC infrastructure of IIT and the Peregrine HPC cluster of the University of Groningen.

\section*{Funding} \label{Funding}
This work has been supported by European Union's Horizon 2020 MSCA Programme under Grant Agreement No 813713 NeuTouch, the “University SAL Labs” initiative of Silicon Austria Labs (SAL) and its Austrian partner universities for applied fundamental research for electronic based systems, the CogniGron research center and the Ubbo Emmius Funds of the University of Groningen.

\section*{Data Availability Statement} \label{Data Availability Statement}
The code for encoding, training, inference and exporting the HDF5 file from PyTorch is open source and available in a \href{https://github.com/event-driven-robotics/tactile_braille_reading}{GitHub repository}. The code for importing the HDF5 into Loihi is shared within the Intel Neuromorphic Research Community (INRC). The \href{https://zenodo.org/record/6556273}{dataset} is open access and uploaded in Zenodo under the DOI 10.5281/zenodo.6556273~\cite{muller_cleve_simon_f_2022_6556273}.

\section{Supplementary Data}
\subsection{Energy consumption approximation from SNN simulation} \label{sup:spikes_inference}
While the energy consumption from the hardware implementations can be directly measured, in simulation it can be approximated by determining the number of spikes in each layer of the network. In Tab.~\ref{tab:snn_spikes} we show the total number of spikes normalized per letter in the dataset. The spikes have been calculated in simulation with the parameters and weights pre-trained according to the Tab.~3. Similar to the profile of energy consumption, as seen in the hardware implementation results, an increase in the threshold of the dataset transformation, that initially decreases the number of spikes in the input layer, does not correspond with a decrease in the number of spikes in the total network.

\begin{table}
\caption{Number of spikes in each of the layers of the network trained with parameters obtained from the optimization. Spikes calculated as the inference of the whole dataset. Percentages as ratio of spikes with the equivalent in threshold 1 as reference.\\}
\label{tab:snn_spikes}
\centering
\begin{tabular}{ccccc}
\multicolumn{1}{l}{} & \multicolumn{1}{l}{} & \multicolumn{2}{c}{\textbf{Delta  coding threshold}} & \multicolumn{1}{l}{} \\ \hline
Recurrent            & \textbf{1}           & \textbf{2}                & \textbf{5}               & \textbf{10}          \\ \hline
\textbf{IN layer}    & 698                  & 426 (61\%)                & 126 (18\%)               & 41 (6\%)             \\
\textbf{MID layer}   & 1230                 & 2165 (176\%)              & 2775 (225\%)             & 2591 (210\%)         \\
\textbf{OUT layer}   & 371                  & 301 (81\%)                & 508 (137\%)              & 257 (69\%)           \\
\textbf{Total}       & 2298                 & 2892 (126\%)              & 3409 (148\%)             & 2889 (126\%)         \\ \hline
Feedforward          & \textbf{1}           & \textbf{2}                & \textbf{5}               & \textbf{10}          \\ \hline
\textbf{IN layer}    & 698                  & 426 (61\%)                & 126 (18\%)               & 41 (6\%)             \\
\textbf{MID layer}   & 1029                 & 4881 (474\%)              & 4917 (478\%)             & 2817 (274\%)         \\
\textbf{OUT layer}   & 496                  & 507 (102\%)               & 833 (168\%)              & 235 (47\%)           \\
\textbf{Total}       & 2223                 & 5815 (262\%)              & 5876 (264\%)             & 3092 (139\%)
\end{tabular}
\end{table}

\subsection{Support-Vector Machine} \label{sup:svm}

We applied a \gls{svm} with a linear kernel. The \gls{svm} was provided with raw sensor readings in the first place. In a pre-processing step, we first subtracted the sensor recordings from 255 to invert its range and make an increasing value represent increasing pressure on the sensor. We then normalised the resulting values by the maximum value out of all sensors and all trials. As last step, the sample-based values for all 12 sensors were concatenated and split in a training and test dataset, using 80\% and 20\% of the samples respectively. To validate the impact of the temporal nature, the \gls{svm} was evaluated with a continuously increasing number of samples, starting with 1 sample. Each iteration included 1 more sample until all samples were included. Each iteration was trained and tested 50 times.

\subsection{Time-Series Classification} \label{sup:tsc}

More details regarding the architectures used for time-series classification:

\gls{fcn}: Mainly a \gls{cnn} but without local pooling layers, which keeps the length of the time series unchanged. The last \gls{fc} layer is replaced by a \gls{gap} layer, helping to identify which of the time series contributed the most to the classification and reducing the number of parameters. The network contains 3 convolutional blocks with a stride of 1, each performing three operations: 1) convolution, 2) batch normalization, and 3) ReLU activation function. The result of the third convolutional block is averaged over the whole time dimension. The first convolution contains 128 filters with a filter length of 8, the second contains 256 filters with a filter length of 5, and the last contains 128 filters with a filter length of 3.

\gls{resnet}: The network is composed of 11 layers where the first 9 are convolutional layers, of which every 3 layers are forming a residual block. Short-cuts between these residual blocks are the main characteristic and difference to \glspl{fcn}. These blocks are followed by a \gls{gap} layer, averaging the time series across the time dimension, and a final softmax layer, containing neurons equal to the number of classes in the dataset. The convolutions in the residual blocks have a filter length fixed to 64 with batch normalization followed by ReLU as activation function. The filter length in the residual blocks is 8, 5, 3 respectively.

Encoder: A hybrid deep \gls{cnn} inspired by \glspl{fcn}, whereas the \gls{gap} layer is replaced by an attention layer. The first three layers are convolutional layers. The first has 128 filters with a length of 5, the second 256 filters with a length of 11, and the third 512 filters with a length of 21. Each convolution is followed by an instance normalization operation and fed to a \gls{prelu}. The output of the \gls{prelu} is followed by a dropout operation with a rate of 0.2 and a final max pooling of length 2. Finally, after the attention layer, a softmax classifier with the number of neurons equal to the classes in the dataset is used.

\gls{tcnn}: The \gls{tcnn} differs in 3 main points from the previously described networks. First, instead of using the cross-entropy loss function, \gls{mse} is used, leading to a \gls{fc} final layer with sigmoid as activation function. Second, local average pooling instead of local max pooling is used and one convolution is applied to each dimension of the \gls{mts}. Third, the final classifier is \gls{fc} directly to the second convolution. Two convolutional layers with filter lengths 6 and 12 are followed by a local average pooling operation with a length of 3 and sigmoid as activation function. The convolutions are applied on all dimensions. The final \gls{fc} layer has neurons equal to the number of classes in the dataset and sigmoid as activation function. Further, the output of the second convolution is directly fed as input to the \gls{fc} layer.

Inception: The network is composed of two residual blocks, each containing three inception modules. Each inception module contains three main parts. The first part is a bottleneck layer, which performs convolution to represent the input according to the user defined size, the default value is 32, to reduce the input dimensionality. The second part are three parallel sliding filters with increasing length of 10, 20 and 40, which are applied to the output of the previous bottleneck layer. The third part is a max pooling layer, performed on the input of the inception module, which is followed by a bottleneck layer. The output of the three filters together with the this bottleneck layer are concatenated and form the output of a module. The two residual blocks are followed by a \gls{gap} layer and a final \gls{fc} layer with neurons equal to the number of classes in the dataset.

\subsection{Parameter counting} \label{sup:lstm}
\subsubsection{LSTM}

The calculation of the number of parameters in the \gls{rsnn} was made according to
\begin{equation*}
    nb\_parameters = nb\_channel \times nb\_input\_copies \times nb\_hidden + nb\_hidden^2 + nb\_hidden \times nb\_output
\end{equation*}
with $nb\_channel = 12$, $nb\_input\_copies = 2$, $nb\_hidden = 450$, and $nb\_output = 27$.

The \gls{lstm} recurrent neural network is comprised of four basic gates that take as input the concatenation of the input feature vector $x_{(t-n)}$ and the previous hidden state, and can be conceptually summarized as

\begin{enumerate}
    \item \textit{Input gate}: it decides which new values are going to be incorporated to the cell.
    \item \textit{Forget gate}: it takes the input $x_t$, and the previous hidden state $h_{t-1}$ and delivers a number between 0 and 1 for each cell state $C_{t-1}$ indicating how much past information should be remembered.
    \item \textit{Estimated cell state}: New candidate vector created also from the input $x_t$ and previous hidden state $h_{t-1}$, that is combined with the information coming from the input gate, to produce the new cell state $C_t$.
    \item \textit{Output gate}: it filters the new cell state and combines it with a filtered version of the input and last hidden state $h_{t-1}$ to produce the new hidden state $h_t$.
\end{enumerate}

Each gate has the same number of trainable parameters in the form of weight $W$ and bias $b$, and can be calculated by hand as

\begin{equation*}
    \text{parameters per gate} = \underbrace{h_{size}\times(h_{size} + x_{size})}_{\text{number of } W_{i}} + \underbrace{h_{size}}_{\text{number of } b_{i}}
\end{equation*}

where $(h_{size} + x_{size})$ is the resulting dimension of the concatenation between the input feature $x_t$ and previous hidden state $h_{t-1}$. The input feature $x_t$ results from replicating $i_{copies}$ times the values of the 12 taxels in each time step, resulting in a total parameter count of

\begin{equation*}
    \text{total number of parameters} = 4\left[h_{size}\times (12 \times nb\_input\_copies + h_{size}) + h_{size} \right] + nb\_output \times h_{size}
\end{equation*}

\begin{figure}
    \centering
    \includegraphics[width=0.75\textwidth]{./figure/fig_12_LSTM}
    \caption{Implemented single layer \gls{lstm} with a fully connected of size $h \times 27$. The green line represents the concatenation operation, the red blocks represent weighted operations followed by the activation indicated in each label, the yellow are element-wise operations, and the blue are $tanh$ activation functions. }
    \label{fig:LSTM_arch}
\end{figure}

In this case,  we aim to have the same number of parameters as the best performing \gls{rsnn} for comparison purposes, leading to 236,892 parameters in total with 228 hidden units.

\bibliographystyle{IEEEtran}