--- abstract: 'During the past two decades, epileptic seizure detection and prediction algorithms have evolved rapidly. However, despite significant performance improvements, their hardware implementation using conventional technologies, such as , in power and area-constrained settings remains a challenging task; especially when many recording channels are used. In this paper, we propose a novel low-latency parallel architecture that has between 2-2,800x fewer network parameters compared to architectures and achieves 5-fold cross validation accuracy of 99.84% for epileptic seizure detection, and 99.01% and 97.54% for epileptic seizure prediction, when evaluated using the University of Bonn , CHB-MIT and SWEC-ETHZ seizure datasets, respectively. We subsequently implement our network onto analog crossbar arrays comprising devices, and provide a comprehensive benchmark by simulating, laying out, and determining hardware requirements of the component of our system. To the best of our knowledge, we are the first to parallelize the execution of convolution layer kernels on separate analog crossbars to enable 2 orders of magnitude reduction in latency compared to hybrid Memristive- accelerators. Furthermore, we investigate the effects of non-idealities on our system and investigate to mitigate the performance degradation due to low / resolution. Finally, we propose a stuck weight offsetting methodology to mitigate performance degradation due to stuck $R_{\text{ON}}/R_{\text{OFF}}$ memristor weights, recovering up to 32% accuracy, without requiring retraining. The component of our platform is estimated to consume approximately 2.791W of power while occupying an area of 31.255mm2 in a 22nm FDSOI CMOS process.' author: - 'Chenqi Li$^\dagger$, [^1] Corey Lammie$^\dagger$, Xuening Dong, Amirali Amirsoleimani, Mostafa Rahimi Azghadi, and Roman Genov, [^2] [^3]' bibliography: - 'References.bib' title: Seizure Detection and Prediction by Parallel Memristive Convolutional Neural Networks --- CNN, Seizure Detection, Seizure Prediction, EEG, RRAM, Memristive Crossbar Array Introduction ============ is a common neurological disorder that affects approximately 1% of the world's population @Beghi2019. A seizure is characterized by excessive firing of neurons in the brain, while epilepsy is a medical condition that involves recurrent seizures @Stafstrom2015. As the underlying occurrence mechanism of epilepsy is not well understood @Patel2019 [@Brennan2018; @Gasparini2019], it requires experimental methods of treatment that rely on accurate detection and prediction systems, as depicted in Fig. \[fig:1\]. ![An overview of a typical epileptic seizure detection and prediction system. Acquired signals are sampled and processed near-sensor using an , prior to being sent wirelessly to edge device(s) for real-time pre-processing and feature extraction. Features can then be fed into and/or architectures, residing either on the edge or in the cloud, which perform epileptic seizure detection and prediction. <span data-label="fig:1"></span>](1.pdf){width="50.00000%"} is the most common method used to monitor the electrical activities of the brain, and can be used to detect and predict seizures. There have been numerous applications of traditional algorithms, such as , , and classifiers to classify ictal (seizure), preictal (prior to a seizure) and non-ictal (non-seizure) signals using recordings. Despite being able to achieve high accuracies, these approaches require the manual extraction and selection of features in the time- or frequency-domain @Siddiqui2020. The optimal choice of such feature extractions are largely unknown, experimental, and dependant on specific patient signatures, such that there is no one-fit-all solution. Compared to traditional seizure classification algorithms, -based algorithms have more advantages in complex signal feature extraction, as they do not require feature engineering, and are capable of outperforming traditional algorithms for epileptic seizure detection and prediction tasks @Ibrahim2022. However, when these systems are implemented using , there are problems such as large scale, high calculation energy consumption and high delay, which hinder their efficacy; especially in resource-constrained environments. In order to solve this kind of problem, this paper proposes a neuromorphic calculation strategy based on a novel architecture, which utilizes analog crossbars. Computer designers have traditionally separated the role of storage and compute units. The paradigm blurs this distinction, and imposes the dual responsibility on memory substrates: storing and computing on data for massively parallel computing @Das2022. By exploiting the physical characteristics of emerging analog device technologies, analog crossbars can be used to perform , the most dominant operation in , in as little as $\mathcal{O}(1)$ @Li2018 [@Lammie2019], significantly reducing the computational complexity during inference operations. Our specific contributions are as follows: 1. To the best of our knowledge, we are the first to parallelize the execution of convolution layer kernels on separate analog crossbars to address the computational bottleneck of , enabling 2 orders of magnitude reduction in latency compared to current hybrid Memristive- accelerators; 2. We reduce the number of required parameters by 2-1,600x and 5-2,800x for epileptic seizure detection and prediction tasks using deep learning models, while still achieving SOTA performance; 3. We provide a comprehensive benchmark for hardware memristor-based seizure prediction/detection systems by simulating, laying out, and determining hardware requirements of the CNN component of our system; 4. We propose a simplified stuck weight offsetting methodology for mitigating severe degradation of system performance due to stuck $R_{\text{ON}}/R_{\text{OFF}}$ memristor weights. We demonstrate that our method is capable of achieving up to 32% performance recovery, without requiring retraining, while incurring minimal hardware and computational overhead. To promote reproducible research, all of our simulation codes are made publicly accessible[^4]. The rest of the paper is structured as follows: In Section \[sec:related\_work\], we overview and discuss related work. In Section \[sec:novel\_system\], we present our epileptic seizure detection and prediction system. In Section \[sec:software\_methodology\], we overview and discuss our software methodology. In Section \[sec:hardware\_simulation\_methodology\], we overview and discuss our hardware simulation methodology. In Section \[sec:results\_discussion\], we present and discuss our results. Finally, we conclude the paper in Section \[sec:conclusion\]. \[!t\] ![image](2.pdf){width="95.00000%"} Related Work {#sec:related_work} ============ In this section, we present an overview of related work using parallel and related work using traditional and neuromorphic architectures for the detection and prediction of epileptic seizures using and signals. Parallel --------- Parallel are composed of one or many convolutional layers, which are executed in parallel and have been previously used in many applications. For example, in the ResNeXt @Xie2017 family of architectures, parallel blocks containing convolutional layers were used to increase network width, which can decrease the time required to train a @Zagoruyko2016. When performing multi-modal , parallel convolutional layers can be used to process different inputs in parallel @Ngiam2011, in order to improve network throughput. Specifically for epileptic seizure detection and prediction tasks, parallel convolutional layers have been used to learn high-level representations simultaneously @Wang2021. By parallelizing convolutional operations, inference time is greatly reduced compared to current SOTA architecture that rely on sequential convolution layers, as convolution layers form the bottleneck of inference. Traditional -based Seizure Detection and Prediction Algorithms -------------------------------------------------------------- As early as 1996, initial attempts were made to detect seizures using signals and traditional approaches. Using a combination of and wavelet transforms, sensitivity values of 76% @Webber1996 and 97% @Pradhan1996, were reported using standardized datasets. In the late 2000s and early 2010s, encountered growing interest. Namely, when using in combination with feature extraction methods such as high-order spectra analysis, wavelet transforms, , wavelet decomposition and least-squares parameter estimators @Chan2008 [@Netoff2009; @Chua2009; @Sorensen2010; @Chisci2010; @Petersen2011; @Temko2011; @Acharya2011; @Kharbouch2011; @Liu2012], promising sensitivity, specificity, and accuracy values $\geq$98.5% were achieved. More recently, advances in the domain using and , have further benefited seizure detection algorithms. Current models are capable of achieving accuracy ranging from 95-100% @Ullah2018 [@Zhao2020; @Abiyev2020; @Boonyakitanont2019] across multiple datasets. Early efforts for seizure prediction started in 1970s, where seizure warning systems were designed with logic circuitry to classify extracted features from a series of filters and analog circuitry @Liss1973 [@Viglione1973]. To varying degrees of success, a variety of methods have been proposed, including a rule-based method using univariate measures @Aarabi2012, spike rate analysis @Li2013, positive zero-crossing intervals analysis @Zandi2013, statistical dispersion measures @Bedeeuzzaman2014, multidimensional probability evolution @McSharry2002, circadian concepts via probabilistic forecasting @Schelter2011, and a combination of reinforcement learning, online monitoring and adaptive control theory @Wang2010. Similarly to seizure detection, many techniques have also been applied. Notable contributions include the combination of and , capable of achieving 99.6% accuracy and a of 0.004 per hour @Daoud2019. Moreover, supervised deep convolutional autoencoder and bidirectional long short-term memory networks have been used to achieve accuracy, sensitivity, specificity, and precision values between 98-99%, with F1-values $\geq$0.98. More recently, augmented network architectures have been used to reduce computational complexity for operation in resource-constrained environments. One such approach, which employs with minimizing channels, is capable of achieving 99.47% accuracy, 97.83% sensitivity, 92.36% specificity, with a of 0.0764 @Jana2021. Finally, Siamese models have been used to achieve 88-91% accuracy on the CHB-MIT dataset @Dissanayake2020. We refer the reader to @Alotaiby2014 for a comprehensive survey of seizure detection and prediction algorithms. Hardware Implementations of -based Seizure Detection and Prediction Algorithms {#litreview} ------------------------------------------------------------------------------ Many hardware implementations of epileptic seizure detection and prediction algorithms have been reported using a variety of technologies; namely , and @Azghadi2020 [@Kassiri2017]. Complementing traditional hardware implementations, architectures, which use memristive crossbar arrays to perform repetitive operations in-memory, have gained increasing popularity in recent years. Kudithipudi *et al.* implemented a neuromemristive reservoir computing architecture to achieve 90% accuracy and Merkel *et al.* achieved 85% accuracy @Kudithipudi2016 [@Merkel2014]. Nature-inspired memristive was implemented by Karamani *et al.* to emulate epilepsy-related phenomena in the brain @Karamani2018. Recent works by Liu *et al.* implemented filter bank on memristive crossbar array to achieve 93.46% seizure detection accuracy and obtained 95% accuracy by using a memristive crossbar based signal-processing stage combined with linear discriminant classifier @Liu2020. Lammie *et al.* pioneered the implementation of for seizure prediction using memristor arrays, achieving 77.4% sensitivity and 0.85 on the CHB-MIT dataset @Lammie2021. Seizure is a chronic, recurring condition that can mostly be prevented through medication before onset @Usman2017, but even with the best medications, 30% of the patients are drug-resistant @Fujiwara2015. Closed-loop brain stimulation has been found to mitigate and even improve symptoms @Zanghieri2021 [@Heck2014], but unpredictability of seizure requires a closed-loop prediction system to provide accurate warning with adequate preparation time for stimulation @Nasseri2021. This calls for the need for fast, low-latency computations, as the changes within the patients can be noticed early-on, in order to start treatments early to improve safety and quality of life @Yang2018. In doing so, symptoms and subsequent effects can be minimized, including anxiety and social exposition @Pinto2021. The major limiting factor of seizure detection and prediction algorithms is the reliance on patient specific features, leading to undesirable results when generalized to other patients in the real world @Stirling2021. With energy efficient computations, it enables the deployment of such systems within wearable devices, so that it can be coupled with the stimulation system, as well as allowing data for a patient to be collected in the long-term to further improve model's predictions by fine tuning the model to better recognize patient-specific signatures @Peng2021. It is known that convolutional layers are the bottlenecks of . According to Cong et al., convolutions make up more than 90% of inference @Cong2014. Therefore, accelerating convolution is pivotal to efficient for future seizure detection/prediction systems. Note that all existing hardware implementations of memristive accelerators focus on sequential . Memristive crossbar acceleration of parallelized convolution layers and blocks, found in many CNN architectures such as ResNeXt @Xie2017, are explored in this work to further reduce inference latency. Seizure Detection and Prediction System {#sec:novel_system} ======================================= In this section, we present our seizure detection and prediction system. As shown in Fig. \[workflowl\], our system comprises of five stages, depicted using Fig. \[workflowl\](a)-(e). As the same network architecture, depicted in Fig. \[workflowl\](f), is used for both detection and prediction, and networks are bench-marked using multiple datasets, our proposed system can be reconfigured for both epileptic seizure and prediction tasks. While we briefly detail and discuss signal acquisition and pre- and post-processing stages, here-on-in, the scope of this paper will be largely confined to the accelerator step described in Fig. \[workflowl\](d). We leave a detailed hardware description and evaluation of other stages to future work. Parallel Convolutional Neural Network Architecture -------------------------------------------------- The primary constraint put on our design was a fixed modular tile size of 64$\times$64. Practically, passive memristor-based analog crossbar tiles of sizes up to 128$\times$64 have been used to perform @Li2018, however such designs have only been demonstrated using pseudo-crossbars having micron-size electrodes. Such limitations in the maximum viable size are a serious computational scalability challenge with electrodes in the tenth of nanometer range that would prevent sinking large currents through them @Amirsoleimani2020. Recently, a 4K memristor analog-grade passive crossbar circuit has been fabricated @Kim2021, which comprises several modular 64 x 64 passive crossbar tiles with 99% functional nonvolatile metal-oxide memristors. From an original exploratory investigation, it was determined that for the device being modelled, the largest feasible modular tile size which is able to be programmed using a write-verify scheme was 64$\times$64. Consequently, this fixed modular tile size was used in our designs to minimize the power and area overhead of peripheral circuits and tile interconnects, which are much larger when smaller fixed modular tiles are used. Model Search and Selection {#sec:modelselection} -------------------------- Most current state-of-the-art employ sequential convolution layers, whereby subsequent convolution operations are dependent on results from previous layers. However, in parallel , convolution layers can be processed simultaneously, enabling the use of multiple crossbars at the same time. In addition, parallel convolution layers with different kernel sizes enable the network to extract features of varying receptive fields, providing the fully connected layers a diverse and yet compact representation of the features for classification; enabling a reduction in network parameters required. \[!t\] Fixed modular crossbar tile size ($m \times n$), $\textsc{Obj}_{max}$, objectives to minimize, $\textsc{Obj}_{min}$, additional hardware design constraints, $\mathbf{w}$. Optimized network architecture $(L, D, \boldsymbol\alpha, \boldsymbol\beta)$, where $L$ is the number of convolutional layer blocks, $D$ is the number of fully connected layers, $\boldsymbol\alpha$ is a vector containing the sizes of the first kernel for each convolutional layer when parallel convolutional layer execution is performed, and $\boldsymbol\beta$ is a vector containing the number of output neurons for each fully connected layer **minimize** $\textsc{Obj}(m, n, L, D, \boldsymbol\alpha, \boldsymbol\beta)$ subject to **w**. $\mathbf{C_{in}}_l$ = $m$ $\mathbf{C_{out}}_l$ = floor($n$ / 2) $\mathbf{k}_{l0} = \boldsymbol\alpha_l$, $\mathbf{k}_{l1} = m - 2 - \boldsymbol\alpha_l$ $\mathbf{k}_l = m - 1$ $\mathbf{m}_d = \boldsymbol\beta_l$ $\mathbf{m}_{D - 1} = 2$ **maximize** <span style="font-variant:small-caps;">Eval</span>(Net) and **minimize** <span style="font-variant:small-caps;">Params</span>(Net), where, Net = <span style="font-variant:small-caps;">Network</span>\_<span style="font-variant:small-caps;">Architecture</span>($m, n, L, D, \boldsymbol\alpha, \boldsymbol\beta$) $\textsc{Obj}_{min}$(Net) \[alg:prog\_train\_alg\] As shown in Fig. \[workflowl\], our proposed architecture consists of two parallel convolution kernels. Algorithm \[alg:prog\_train\_alg\] formalizes the methodology used to search for and select the employed model. For our selected model, latency was minimized using $\textsc{Obj}_{min}$. $L$, $D$, and $\boldsymbol\beta$ were fixed to values determined empirically using a preliminary exploratory analysis, and $\boldsymbol\alpha$ was optimized as per Algorithm 1. The following additional hardware design constraints were imposed for our design: all convolutional layers must be capable of fitting onto one modular crossbar tile, and the total number of required modular crossbar tiles must not exceed 8. As the convolution operation bottlenecks inference, the size of kernels used in parallel convolution layers need to be carefully considered to optimize both network performance and latency. In our proposed architecture, shown in Fig. \[workflowl\](f), we have two parallel convolution layers and one average pooling layer, comprising one convolutional block. To parallelize the two convolution layers, it would be necessary to map the weights of the two convolution layers onto two separate crossbars. As a design choice, we wanted to retain the flexibility of mapping both convolution layers onto the same crossbar, if space complexity is prioritized over latency. Therefore, during the kernel size search, we imposed a constraint of 62, i.e., $m - 2$, for the sum of convolution kernels, as 2 additional rows are designated for implementing the bias for both parallel convolution layers. When denoting the kernel size of the first parallel convolutional layer as $\alpha$, the kernel size of the second parallel convolutional layer can be expressed as $62 - \alpha$. To determine the optimal network architecture, the University of Bonn's seizure dataset @Andrzejak2001 was used. Specifically, a 80:20 train validation split was employed, and <span style="font-variant:small-caps;">Eval</span>(Net) was used to determine the 5-fold cross validation accuracy. Seed values of 32 and 8 were arbitrarily set for the network architecture search, to ensure reproducibility of results, and to reduce bias between search and validation. Empirically, $L=1$, $D=2$, and $\boldsymbol\beta$=\[8,\] achieved substantial performance. For the single convolutional block, $\boldsymbol\alpha_0$ was varied between 31 and 60. A validation accuracy of 100% was achieved for all values of $\boldsymbol\alpha_0$, except for $\boldsymbol\alpha_0=60$, which achieved an optimal validation accuracy of 99.375%. This is not surprising, as the window size of input data is only 64. Therefore, convolution kernel sizes of 60 and 2 provides two extreme and dramatically different receptive fields. In particular, a kernel size of 2, which corresponds to around 10ms of melissa85@example.net, is likely insufficient to capture local correlation and learn seizure characteristics. The final model was chosen using Occam's razor principle, whereby the simplest model is the best model. Consequently, a kernel size of 32 was selected, as a kernel size 31 would be the simplest to implement due to symmetric convolution kernel sizes; however 32 provides a more diverse receptive field. To further demonstrate the advantage of varied kernel size, a 5-fold cross-validation was performed using a) 64 filters of kernel size 31 b) two parallel convolution layers each with 32 filters of kernel size 30 and 32 (see Fig. \[workflowl\]). It was observed that both networks are capable of achieving accuracy varying between 99.61% to 99.83%, but varied kernel size leads to +0.03%, -0.01%, +0.02% change in performance on Bonn, SWEC-ETHZ and CHBMIT datasets, respectively, compared to using 64 filters of kernel size 31. Although a small degradation in performance is observed for SWEC-ETHZ dataset, improvements are observed for both Bonn and CHBMIT dataset. A net improvement is observed for both seizure detection and prediction using a varied kernel size, while both experiments employ an identical number of weights. ![Architecture hierarchy of our memristive DL accelerator with (a) TDM and (b) Parallelized Implementation.<span data-label="fig:block_diagram"></span>](3.pdf){width="50.00000%"} Hardware Architecture Hierarchy ------------------------------- In Fig \[fig:block\_diagram\], we present our hardware architecture hierarchy. The processing engines comprises 7 memristive crossbar array tiles, as well as I/O registers, eDRAM buffers, and peripheral circuits for ReLU, subtract, and average pooling. We present two configurations for our tile, , and parallelized. In the case, each tile contained a S+H and an for reading out column currents, and one per row for reading inputs in parallel, as shown in Fig. \[fig:block\_diagram\](a). In the parallelized case, each tile contains 64 , as shown in Fig. \[fig:block\_diagram\](b). Software Methodology {#sec:software_methodology} ==================== To train and evaluate our epileptic seizure detection and prediction system, we benchmarked our system using one epileptic seizure detection task and two epileptic seizure prediction tasks. For epileptic seizure detection, the University of Bonn's seizure dataset @Andrzejak2001 was used. For epileptic seizure prediction, the CHB-MIT Scalp @Goldberger2021, and the long-term SWEC-ETHZ @Burrello2019 datasets were used. To perform epileptic seizure detection and prediction, and samples can be categorized as either ictal, interictal or preictal. Ictal samples indicate the presence of a seizure, interictal samples are periods between seizures, and preictal samples can be used to detect the onset of a seizure. For epileptic seizure detection, binary classification is performed between ictal and interictal samples. For epileptic seizure prediction, binary classification is performed between preictal and interictal samples. For both epileptic seizure detection and prediction tasks, on account of unbalanced classes, 5-fold cross validation was used to train and validate our network architecture. Training and Evaluation Methodologies ------------------------------------- ### Epileptic Seizure Detection {#sec:seizuredetec} The University of Bonn's seizure dataset is comprised of 5 sets (A-E), where set A is normal with open eyes, set B is normal with closed eyes, set C and D is seizure free intervals, and set E is seizure only activities. Each set contains 100 single-channel time series of 23.6 seconds, with 4,096 samples in each time series. All data were melissa85@example.net Hz, at a resolution of 12 bits. To perform binary classification between ictal and interictal samples, all samples from sets A and E were used. Both sets (A and E) were divided into samples of 64 seconds periods and randomly shuffled. No augmentation and pre-processing techniques, such as normalization, were performed, as are capable of automatic feature extraction from time-series data and are robust to noise. The lack of need for pre-processing steps implies reduced hardware complexity to perform such operations. Using the network model (with optimal kernel sizes determined in Section \[sec:modelselection\]), a 5-fold cross-validation strategy was used to determine network's performance. To determine performance, the mean of left out set accuracy, sensitivity, specificity, false-positive rate and the across folds of 5-fold cross-validation were reported. ### Epileptic Seizure Prediction {#sec:seizurepred} The CHB-MIT Scalp , and the long-term SWEC-ETHZ datasets were used. The CHB-MIT Scalp dataset comprises of 23 cases, which were collected from 22 subjects (5 males, ages 3–22; and 17 females, ages 1.5–19). The last case was obtained 1.5 years after the first, from one of the female subjects @Goldberger2021. All signals were sampled at 256Hz with 16-bit resolution, using 23-26 electrodes. During data acquisition, no augmentation steps were performed. $^*$Hours. $^\dagger$Samples. \[tab:prediction\_overview\] ![Depiction of (a) our adopted overlapped sampling technique extracting $n$ samples from a continuous preictal segment, and (b) the and terms. As can be seen, continuous preictal segments are extracted during the . All preictal samples that occur during the period are discarded.<span data-label="fig:pre_prediction"></span>](4.pdf){width="45.00000%"} The long-term SWEC-ETHZ dataset comprises of 18 patients with pharmaco-resistant epilepsy, who were evaluated for surgery at the of the University Department of Neurology at the Inselspital Bern @Burrello2019. All signals were sampled at either 512Hz or 1025Hz with 16-bit resolution, using 26-100 electrodes. During data acquisition, after analog-to-digital conversion, a digital band-pass filter was used to filter signals between 0.5 and 150Hz using a fourth-order Butterworth filter. Moreover, forward and backward filtering was applied to minimize phase distortion. Due to computation burden of crossbar simulation, we report the performance using the first 5 viable cases of the the CHB-MIT Scalp and long-term SWEC-ETHZ datasets, reducing the computation required, similar to @Truong2018 [@Wang2021]. In Table \[tab:prediction\_overview\], we present an overview of all cases used to perform binary classification between preictal and interictal samples. A case was categorized as viable if it contained valid labels (namely time-stamps) and data files (i.e., no recording files were missing or corrupt). For both datasets, the first 22 channels of each patient were extracted and used. All signals were down-sampled to 256Hz, and a window size (batch size) of 64s was used when extracting samples. After discarding seizures that occur in the first 20-minute monitoring period, a of 30m and a of 5m were used to extract and label preictal samples for all cases; both of which have previously demonstrated significant performance @Truong2018. These terms are defined visually in Fig. \[fig:pre\_prediction\]. Interictal samples were extracted from one hour recording segments containing no seizures (ictal samples) to reduce class inbalance during training. Next, 176 features per sample were extracted (8 per channel per window/batch interval): the mean, variance, skewness, kurtosis, coefficient of variation, median absolute deviation of amplitude and , and the shannon entropy. Since the input size of the proposed network is 64, the dimensionality of the input data needed to be reduced. A correlation analysis was first performed across the 176 extracted features, but no particular channel could be removed as no strongly correlated channels were discovered. Using , linear dimensionality reduction via enabled the projection of data to lower dimensional space of 64 principal axes. During training, synthetic preictal samples were generated using an overlapped sampling technique inspired by @Alotaiby2014, by sliding a 64s window with a stride of 32s across continuous preictal segments extracted during the period, as depicted in Fig. \[fig:pre\_prediction\]. The same cross-validation training and evaluation strategy and metrics as described in Section \[sec:seizuredetec\] was employed. Hardware Methodology {#sec:hardware_simulation_methodology} ==================== In this section, we discuss our device technology selection, memristor crossbar array implementations of , and present our adopted hardware simulation methodology. Device Technology Selection --------------------------- Computing with charge-based computing devices is attractive due to their technological maturity, even though they have a relatively large area footprint even at advanced technology nodes and face severe scaling challenges @Sebastian2020. Resistance-based memory, in contrast, can be scaled to the nanometer scale, and has the potential of forming cross-point structures without using access devices, achieving ultra high density. devices are used in our design, as they are widely considered to be the most promising emerging resistance-based memory technology- they operate faster than , have a simpler and smaller cell structure than and devices, and are made of materials that are common in semiconductor manufacturing @Sebastian2020. Memristor Crossbar Array Implementations of Parallel CNNs --------------------------------------------------------- Consider the conductance values of a crossbar array as a matrix and input voltages to a crossbar as a vector. The output current from the crossbar, determined using Kirchoff's and Ohm's Law represents the result of the . Such operations form the core of . Being able to accelerate and parallelize them would facilitate the real-time operation of deeper and heavier neural networks for epileptic seziure detection and prediction in resource-constrained hardware @RahimiAzghadi2020. ![A comparison of possible mapping schemes. (a) visualizes the staggering mapping of convolution weights, which is commonly adopted due to its ability to produce all results within a single pass through the crossbar array. (b) visualizes our proposed mapping scheme, without staggering of convolution weights and sparsity in crossbar, at the cost of increased read/write operations. (c) provides a comparison of methods (a) and (b), visualizing when one method should be chosen over the other. <span data-label="mappingcompare"></span>](5.pdf){width="50.00000%"} \[!t\] ![The crossbar parameter mapping layout adopted. Seven 64 $\times$ 64 modular crossbar tiles are utilized. Bias terms of fully connected layers, and the single pooling layer, `pool1`, are computed using additional digital circuitry. To reduce the number of unused devices, parameters of different layers are shared between tiles.<span data-label="workflow2"></span>](6.pdf){width="50.00000%"} To represent signed weight matrices on memristive crossbar arrays, as negative conductance values cannot be expressed using analog memristive devices, a differential mapping scheme was adopted, where two columns of memristors are chosen to represent positive and negative weights, respectively. The signed output is thus the arithmetic difference of current from both columns. In the case of 1D , fully connected and convolutional layers can be decomposed into a series of dot products between inputs, represented as voltages, and weights, represented as memristive conductance. For convolutional layers, the `im2col` algorithm @Chellapilla2006 can be used to map convolutional kernels onto separate crossbar columns. With a single pass, $m$ 1D convolutions can be performed simultaneously, where $m$ represents the number of columns. Average pooling and ReLU operations are performed using additional digital circuitry. Hardware Simulation Methodology ------------------------------- Based on existing literature from Section \[litreview\], all mapping of convolution kernels onto crossbars are sparse, whereby the convolution kernels form a sparse diagonal matrix, as depicted in Fig. \[mappingcompare\](a). This naive approach is extremely space demanding, as the kernels are staggered multiple times throughout the crossbar array, rendering a lot of memristive cells unused. To reduce the space requirement of mapping scheme (a), one possible approach is to build upon the input-stationary concept. One may remap the crossbar weights during inference and replace them with different kernel weights, while reusing the input fetched from memory. On the other hand, one may build upon the weight-stationary concept, as depicted in Fig. \[mappingcompare\](b). In this scheme, convolution kernels can be mapped without staggering before inference. For kernels to convolve against different parts of the signal, the input signal slides. The bottleneck of this approach now lies within fetching input data, requiring additional read/write operations on the peripheral of the crossbar compared to mapping scheme (a). The weight-stationary approach is more efficient compared to the input-stationary approach, as crossbar weight writes can be very time and energy consuming, compared to fetching of inputs and staggering them with shifting circuitry. Fig. \[mappingcompare\](c) provides visualization of when one scheme should be adopted over the other. \[!t\] \[tab:softwaresummary\] A comparison of the naive approach and our proposed weight-stationary approach is performed for our network architecture in Table \[tradeoff\_table\]. As can be observed, the number of memristor cells required for scheme (b) (depicted in Fig. \[mappingcompare\] (b)) is significantly smaller, due to the compact nature of the mapping. This comes, however, at the cost of 33x increase in computation. When taking sparsity, i.e. unused memristors depicted by the gray background in Fig. \[mappingcompare\] (a), into consideration, scheme (b) demonstrates even more significant reduction, i.e. 63x-73x fewer memristors required, while the computation increase remains constant. Unlike convolutional layers, fully connected layers do not involve sliding of signals, so for fully connected layers were implemented using the naive scheme (a). Using scheme (b), we mapped convolutional kernels within our trained network onto crossbars tiles of 64$\times$64. While scheme (b) was chosen for our hardware design, if scheme (a) were chosen with different $n_{ker}$ and $l_{ker}$ values, or the added space complexity is not of concern, the staggered weights of scheme (a) would enable all rows of the crossbars to be employed simultaneously. By choosing the input size of our network to be 64, we maintain the flexibility of mapping with scheme (a) to make use of all crossbar rows simultaneously. As Fig. \[workflow2\] demonstrates, for parallel convolution layers to be accelerated simultaneously, it was necessary to map the weights of the `conv1` and `conv2` onto two separate crossbar tiles. The weight of the `fc1` layer is a matrix of 1088$\times$8, and using a differential weight scheme, would require 1088$\times$16 memristors. The weight matrix can be further divided into 17 sections of 64$\times$16 weights. To maximize the usage of each 64$\times$64 crossbar array, 4 sections of 64$\times$16 weights can be stacked horizontally onto each crossbar, requiring a total of 5 crossbar tiles. Since there are unused memristors on the convolution tiles and `fc2` layer operations are not performed immediately after convolution operations, we decided to map the weights of `fc2` onto the convolution layer tile, instead of using another tile. Note that since the simulation serves as a validation for proof-of-concept, we decided to use the same dimensions for all 7 crossbar tiles. We do recognize that tile 1, 2 and 7 have many unused memristor devices, as a result, performing small VMMs on a large switch matrix. This leads to large power overhead due to high amortized ADC/DAC power over a small matrix and charge/discharge of long row and column wires without using full length for computation. To address such problem in a real medical device, instead of using square tiles, tile 1, 2 and 7 can be easily mapped onto rectangular tiles of the exact required dimensions. Impact of Device and Crossbar Non-Idealities -------------------------------------------- Memristors and memristive crossbar arrays are prone to numerous device and circuit non-idealities which have been demonstrated to severely impact the performance of memristive accelerators @Krestinskaya2019. Consequently, they should be comprehensively simulated prior to circuit-level realization. In this paper, preliminary simulations were performed using the MemTorch @Lammie2020 simulation framework, and comprehensive simulations of the system using passive crossbar arrays were performed using the crossbar array model provided by @Chen2013. Non-idealities considered include input and output resolutions, weight write resolution, weight write deviation, stuck $R_{\text{ON}}/R_{\text{OFF}}$ devices, line and source resistance, and conductance range variation. Other memristive phenomena, such as the dynamic behavior of switched memristive neural networks after programming @Cheng2021, and read disturbance @Shim2020, are not accounted for, as practical metal-oxide memristors are endurance-limited, during programming a write-verify scheme is used, and during inference, all voltages are constrained to have a maximum absolute amplitude of 0.3V @Shim2020. Stuck Weight Offsetting Methodology ----------------------------------- Stuck $R_{\text{ON}}/R_{\text{OFF}}$ weights are known to cause significant network performance degradation in memristive crossbar arrays. Existing works have demonstrated performance recovery through a variety of techniques. In 2014, Kannan et al. took inspiration from SRAM/DRAM technologies and repaired crossbar defects using redundant rows and columns @kannan2014detection. In 2017, Liu et al. proposed to identify significant weights before applying a retraining and remapping algorithm @liu2017rescuing. In 2018, Xia et al. proposed a mapping algorithm with inner fault tolerance to leverage the differential mapping scheme of crossbar arrays to tolerate faults @xia2017stuck. In 2019, Zhang et al. proposed the use of matrix transformations to reduce the magnitude of error introduced by stuck-at-fault devices @zhang2019handling. Also in 2019, Yeo et al. modified conventional transimpedance amplifiers to detect when abnormal current is detected at a particular column due to stuck-at-fault devices and repair by retraining the network with the known defects @yeo2019stuck. Among those works, significant hardware or software overhead is introduced through rewriting and tuning of weights, retraining of networks or using additional circuitry. To minimize the overhead, we propose stuck weight offsetting, which improves upon the inner fault tolerance method. Inner fault tolerance first identifies all available (non stuck-at-fault) devices and initializes them to default values. Then, the scheme goes through all available devices and adjusts each value such that the represented values cannot be made any closer to the target matrix parameter. Intuitively, this serves to minimize the incorrect contribution of the $R_{\text{ON}}/R_{\text{OFF}}$ weight. We propose to bypass the initialization of available devices to default values and to focus on the complementary weight of stuck-at-fault devices only. Before writing any weights to the crossbar, all stuck-at-fault devices are identified. For each stuck-at-fault device, if the complementary weight is not stuck-at-fault, we calculate its complementary weight to minimize the difference between represented value and target value. All calculated values, along with normal weights, are then written onto the crossbar. This modification reduces overhead by two means. First, all crossbar weights are only required to be written once, as opposed to twice in the inner fault tolerance method (from default to adjusted). Second, our method focuses on complementary weights for stuck-at-fault devices only, as opposed to all available devices for all target parameters. This method incurs minimum additional computational cost, and does not require retraining. $^*$Not reported. \[!t\] <span>lccrrrrr</span> **Paper** & **Method** & **Parallelized** & **Parameters** & **Sensitivity (%)** & **Specificity (%)** & **Accuracy (%)** & **FPR$^\dagger$**\ \ @Truong2018 & 2D-CNN & & N/R$^\diamond$ & 81.20 & N/R$^\diamond$ & N/R$^\diamond$ & **0.16**\ @Wen2018 \* & 2D-CNN & & N/R$^\diamond$ & N/R$^\diamond$ & N/R$^\diamond$ & 92.00 & N/R$^\diamond$\ @Hossain2019 & 2D-CNN & & 49,560 & 82.71 & 88.21 & 98.19 & N/R$^\diamond$\ @Cao2019 \* & 2D-CNN & & N/R$^\diamond$ & 88.80 & 88.60 & 88.70 & N/R$^\diamond$\ @Tian2019 \* & 3D-CNN & & 28,459,615 & 96.66 & 99.14 & 98.33 & N/R$^\diamond$\ @Liang2020 \* & 2D-CNN & & 9,695,012 & 84.00 & 99.00 & 99.00 & 0.2\ @Wang2021 & 1D-CNN & & 105,538 & 95.55 & **99.68** & **99.64** & N/R$^\diamond$\ Ours & 1D-CNN & & **10,778** & **99.24** & 98.68 & 99.01 & 0.47\ \ @Burrello2020 \* & Ensemble HD & & N/R$^\diamond$ & 96.38 & 97.31 & 96.85 & N/R$^\diamond$\ @Wang2021 & 1D-CNN & & 105,538 & 94.57 & **99.86** & **99.81** & N/R$^\diamond$\ Ours & 1D-CNN & & **10,778** & **98.22** & 97.02 & 97.54 & **0.99**\ Indicates the results are reported across the entire dataset and patient-wise performance was not reported. $^\dagger$False positive rate (per hour). $^\diamond$Not reported. ![image](7.pdf){width="85.00000%"} Quantization Aware Training for Lower Resolution Systems -------------------------------------------------------- A high resolution system is often not feasible to deploy on edge devices, given power consumption constraints and sampling frequency requirements, which are fundamental tradeoffs for resolution in and . However, lower resolution systems with improved power and frequency performance can exhibit performance degradation. This effect was observed for some patients, and more details can be found in Section \[sec:QAT\]. For significant performance degradation (a degradation of 5% or more compared to full resolution system), we propose to perform prior to mapping the weights onto memristive crossbar arrays @Brevitas. During , we quantized the convolutional and fully connected layers of the network to the resolution equivalent to or even lower than that of the resolution of the crossbar weights and / resolution. Quantized layers are implemented using the Brevitas library @Brevitas, which provides PyTorch-compatible convolution and fully connected layers of specified weight resolutions. In addition, inputs to the network were quantized, while intermediate outputs remained not quantized. Network architecture and other training parameters remained unchanged. Results and Discussion {#sec:results_discussion} ====================== Prior to the investigation of device and crossbar non-idealities, we report baseline software results for epileptic seizure detection and prediction using our network architecture, in Table \[tab:softwaresummary\]. 5-fold cross-validation was performed using a different seed to eliminate bias on the first fold. To demonstrate the generalizability of the designed network to different domains and patients, the same architecture was applied for seizure detection and prediction. Unlike the Bonn dataset, both the CHB-MIT and SWEC-ETHZ datasets are multi-channel datasets with larger memory and computation requirements within the time domain. In order to reduce the time and memory complexity, pre-processing steps as described in Section \[sec:seizurepred\] were applied to transform the dataset into frequency domain. The shown results suggest that the proposed network is sufficient and can generalize well for both detection and prediction. Comparisons Against SOTA Software Implementations ------------------------------------------------- In Tables \[comparison\_SOTA\_detection\] and \[comparison\_SOTA\_prediction\], we compare our baseline software implementations that use full precision (32-bit) floating-point parameters against other software implementations in literature for epileptic seizure detection and prediction, respectively. As shown in the Tables, for epileptic seizure detection we achieve performance in 3/4 criteria, while for prediction we obtain SOTA performance in 3/6 criteria. Specifically, for detection, our network architecture is able to achieve an accuracy of 99.84% across all samples without any pre-processing steps, while requiring only 10,778 parameters. This is $\sim$2x fewer parameters than the smallest model in @Ullah2018, which achieved a slightly higher accuracy of 99.90%, while employing various pre-processing steps. Except for the model used in @Abdelhameed2021, which achieves a 100% accuracy, but requires over 10M parameters, all the other models shown in Table \[comparison\_SOTA\_detection\], achieve lower accuracy values despite significantly higher number of network parameters. \[!t\] ![image](8.pdf){width="100.00000%"} For epileptic seizure prediction, pre-processing is performed. Across both datasets, our network architecture achieves the highest sensitivity while requiring the fewest number of parameters. We report close specificity and accuracy values to @Wang2021, which has also used a 1D-CNN architecture with parallelization, but needs $\sim$10x more parameters. Finally, we report the highest across both datasets, however, unlike previous works, we performed no post-processing steps, which may cause this. Also, only two out of the nine previous works have reported their , which makes the comparison incomplete. When mapping trained parameters to ideal crossbars with fully analog devices without any device or circuit non-idealities, the same results were achieved. Generalization Between Datasets ------------------------------- To determine whether or not our trained networks have the ability to generalize, we evaluated the performance of networks trained using the CHB-MIT dataset on the SWEC-ETHZ dataset, and vice-versa in Fig. \[fig:transfer\_learning\]. In addition, we report the cross validation accuracy for networks which have been retrained using transfer learning. To perform transfer learning, parameters were frozen for all layers except the last two fully connected layers, and the weights and biases of the last two fully connected layers were re-trained using the training set of the evaluation dataset. Direct evaluations to/from either of these datasets and the University of Bonn dataset were not made, as the University of Bonn dataset is used for epileptic seizure detection and not prediction, and it is structured differently. Quantization-Aware Training {#sec:QAT} --------------------------- To demonstrate the effectiveness of , we evaluated the performance of our network architecture when trained with and without . Comparisons are made in Fig. \[fig:qat\]. During training, inputs and network weights were reduced to 6-bit resolution, while network architecture and other training parameters were held constant, as described in Fig. \[workflowl\](f). The accuracy, sensitivity, specificity, AUROC, and metrics were all reported and compared. When using 6-bit and , it can be observed that for all patients and metrics, except for specificity of patient 5 from the CHB-MIT dataset, network yields significant performance improvements. \[!t\] ![image](9.pdf){width="100.00000%"} Effects of Non-Idealities on System Performance ----------------------------------------------- Fig. \[nonidealities\_result\] provides a summary of the impact of non-idealities on our system for epileptic detection and prediction. For the University of Bonn dataset, as samples between patients are not explicitly distinguished, the mean and standard deviation of test set accuracy is reported across samples using five arbitrarily chosen seed values. For the CHB-MIT and SWEC-ETHZ datasets, the mean and standard deviation of test set accuracy is reported across samples for the first five viable patients of each dataset, respectively. Across datasets, some patients were observed to be more robust to non-idealities than others. This was observed in our investigations for patients 1, 2, 3 from the SWEC-ETHZ dataset, and patient 2 from the CHB-MIT dataset, for which non-idealities have minimal impact. For the rest of the patients, however, no clear pattern was established with regards to robustness against non-idealities. We attribute the varying degree of effectiveness between patients to underlying patient specific signatures. \[!t\] <span>$^*$The latency is listed as individual element. $^\dagger$S+H = Sample and Hold, IR = Input Register, OR = Output Register.</span> Stuck Weight Offsetting ----------------------- As observed in Fig. \[nonidealities\_result\](d), stuck $R_{\text{ON}}/R_{\text{OFF}}$ devices lead to severe performance degradation. At 1% stuck-at fault and above, system performance can drop below 50% accuracy, rendering the system ineffective. In response to such degradation, we apply our proposed simplified stuck weight offsetting method. Comparing Fig. \[nonidealities\_result\](h) against (d), it is evident that the stuck weight offsetting method improves the average accuracy across all stuck device percentages and datasets. At 1% stuck-at fault, the average accuracy improved by as much as 20% for the Bonn dataset and more than 10% for SWEC-ETHZ and CHB-MIT. The largest improvement was found for the CHB-MIT dataset at 5% stuck-at fault, improving accuracy by 32.11%. At higher stuck device percentages, reduced accuracy recovery is observed. This can be explained by the fact that at higher stuck device percentages, more network information cannot be recovered. Minimizing the contribution of stuck weight cannot fully retrieve the missing information, thereby leading to reduced accuracy recovery. In addition, the proposed method greatly reduces the standard deviation across patients and seeds, thanks to reduced contribution of stuck $R_{\text{ON}}/R_{\text{OFF}}$ devices to final output. The limitation of this method lies within its inability to deal with both elements of the complementary weight being stuck $R_{\text{ON}}$ and $R_{\text{OFF}}$ simultaneously. If a positive (negative) weight is stuck $R_{\text{ON}}$ and negative (positive) weight is stuck $R_{\text{OFF}}$, stuck weight offsetting cannot provide any further adjustment to minimize the error. Meanwhile, if both weights are stuck $R_{\text{ON}}$ or $R_{\text{OFF}}$, the lost weights cannot be recovered, contributing nothing to the final output. \[!t\] <span>$^*$Reported power, area, and latency requirements include the analog front end/signal acquisition component. $^\dagger$Reported power, area, and latency requirements include feature extraction component(s). $^\diamond$Denotes whether systems are able to perform epileptic detection and/or prediction. $^\circ$Not reported.</span> Power, Area, and Latency Requirements ------------------------------------- The following assumptions, all supported by accelerators, are made when estimating the power, area and latency requirements of our proposed memristive accelerator depicted in Fig. \[fig:block\_diagram\], targeting a 22nm process with device integration at the . A memristive device has a fixed area of $100\times100$ nm$^2$ @Lv2021 [@Yu2021] and the device read latency is $6$ ns @Zidan2018. An ADC operating frequency is $10$ MHz @Zidan2018, with a power consumption of $10$ mW @Zidan2018 and a device area of $1.1\times0.6$ mm$^2$ @Yoshioka2005 [@Yu2021]. A operating frequency is $1.25$ GHz, with per unit power consumption of $6$ mW and a device area of $0.0576$ mm$^2$ @Jung2008. Other peripheral circuitry with different purposes, including the activation function @Giordano2019, average pooling layer made up from 4-to-1 multiplexers @Li2016 [@Gaillardon2011], Sample and Hold (S+H) @Shafiee2016, subtractor @Govindarajan2020, and adder @Ganesan2015 circuits, were listed with more detail in Table \[table:power\_area\_latency\_requirements\]. All the peripheral components are scaled to 22nm technology by factors introduced in @Sarangi2021 and all buffers with their associated connections have energy, area and latency estimated by CACTI 7.0 @CACTI7. For all calculations, the source resistance and line resistance of $20$ $\Omega$ and $2$ $\Omega$ are used respectively. To account for RC delays within crossbars when signals are propagated, the methodology presented in @Dozortsev2018 was used, with $C_{SA}$, $T_{settling}$, and $C_{write}$ parameters from @Xu2011. The largest total device latency was used for all devices. In Table \[table:power\_area\_latency\_requirements\], four scenarios are considered: two where the resistance of all active (utilized) devices was fixed to $\bar{R_{\text{ON}}} \approx 10$ $k\Omega$, while considering either or parallel use of , and two where the average resistance of all active devices was assumed to be $(\bar{R_{\text{ON}}} + \bar{R_{\text{OFF}}})/2 \approx 55$ $k\Omega$, again for either or parallelized . These resistance values are representative of two weight distributions: uniform, where all weights are zero, and normal, where all weights are centered around zero. The first distribution was used to report the maximum possible power consumption of our system, and the second distribution was used to report the power consumption of a typical trained using L2-regularization. Considering the marginal impact on total power consumption, ( 0.16% and 0.06% for and parallelized configurations, respectively), the power of each individual trained was not determined or reported. For all scenarios, constant melissa85@example.net per cell @Shim2020 was assumed. Neither crossbar tiles nor peripheral circuitry was assumed to be stacked vertically. Consequently, the circuit area consumption was computed as the summation of all individual elements. Both and were assumed to operate at 6-bit resolution, as stated in Section \[sec:QAT\], for the best performance with . As can be observed in Table \[table:power\_area\_latency\_requirements\], implementations consume significantly less power than parallelized implementations due to the smaller number of required . For the worst case scenario, i.e, when all active devices are programmed to $R_{ON}^-$ with a constant 0.3V read voltage, our proposed memristive accelerator has a latency of $445.22$ $\mu$s, and consumes approximately $2.79$W and $31.255$ mm$^2$ of power and area. This is fairly low power consumption for a accelerator to reside on a separate chip from the neural implant, whereby the implant uses thermal energy to wirelessly communicate with the accelerator @Elansary2021, for reduced latency. It is noted that we have chosen to optimize the latency of our system at the cost of higher power consumption for multiple reasons. Firstly, analog crossbars which are used to perform operations, in particular , require peripheral circuitry which is power- and area-hungry. Consequently, independent of the latency of the system, when inference is being performed, a large proportion of the total system's area and power is consumed by peripheral circuitry, registers, and buffers. While can be used to reduce the total power consumption by increasing latency, other peripheral circuits, registers, and buffers, are still required for operation. Counterintuitively, in certain instances, the energy of the system can be reduced by minimizing system latency during active operation. In other instances, the performance of the system can greatly be improved at the cost of increased power consumption. Secondly, devices suffer from conductance drift induced by read disturbances, which may aggregate, as the analog current is summed up along each during inference @Shim2020. To mitigate this behavior, we have constrained the absolute amplitude of voltages to 0.3V and minimized the duration in which a voltage is applied to each device, i.e., latency is minimized to avoid read disturbances, and to prolong the lifespan of devices, at the cost of increased power consumption. Lastly, as devices are non-volatile, gating circuitry can be used to reduce the energy consumption of both and parallelized architectures, as both of our architectures have a critical delay path which is much shorter than typical signal acquisition sampling rate periods. This also allows for input buffering to be performed, so that constant operation is not required. Comparison to Existing Hardware Implementations ----------------------------------------------- In Table \[table:comparison\_literature\], we compare the performance of hardware implementations of notable epileptic seizure detection and/or prediction hardware systems in the literature. As many different evaluation tasks were used, we did not report performance metrics. Hardware implementations are broadly categorized as either - or -based. As can be observed, both of our implementations (reported for the $(\bar{R_{\text{ON}}} + \bar{R_{\text{OFF}}}) / 2$ scenario in Table \[table:power\_area\_latency\_requirements\]) have significantly reduced inference latency, at the cost of higher power consumption, compared to traditional and -based implementations. It is worth noting that, most of the previous designs have not reported a complete power consumption analysis, are not capable of seizure prediction, and use fewer channels, which can lead to lower power consumption and silicon area. While our proposed system is not currently competitive in resource-constrained environments, it is intended to be used as a reference design for future works implementing epileptic seizure detection and prediction systems using and memristors. Using analog , vertical stacking of crossbars and CMOS components, and partial sensing approaches, the power and area requirements of our simulated system could be greatly reduced. We aim to investigate these in our future research. Conclusion {#sec:conclusion} ========== We proposed a parallel architecture that can be used to perform both epileptic seizure detection and prediction rapidly. Compared to other works in literate, our architecture requires significantly fewer parameters, and demonstrates competitive performance on the University of Bonn, CHB-MIT, and SWEC-ETHZ datasets. Using emerging memristive devices and software-hardware optimization methodologies, we demonstrated, through comprehensive simulations, that our memristive accelerator is capable of performing real-time operation, and consuming reasonable power in
{'source': 'arXiv_out/2206'}