{'fraction_non_alphanumeric': 0.030340216655382533, 'fraction_numerical': 0.010705822613405552, 'mean_word_length': 4.643027698185291, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Improving the Performance of Brain-Computer Interface using Deep Learning Algorithms and Event-Related Spectral Perturbation

Brain Computer Interface allows disabled people to communicate with the external world by using their brain signals. The main goal of a BCI is to provide patients who suffer form any neuromuscular disorders whith a communication channel based on their brain signals. In this paper, the aim is to explore the effects of applying deep learning algorithms and Event Related Spectral Perturbation analyses on the performance of different EEG-based BCI paradigms. Two paradigms were investigated: one is based on the Matrix paradigm (known as oddball); and the other one utilizes the Rapid serial visual Presentation (RSVP) for presenting the stimuli. Deep learning algorithms of convolutional Neural Network (CNN) and Recurrent Neural Network (RNN) were utilized to evaluate the two paradigms. Our findings showed that Matrix paradigm is more effective in detecting P300 signal. In terms of classification methods, deep learning of CNN algorithm has shown superiority performance in comparison with the other machine learning algorithms.


INTRODUCTION
BCI is a direct pathway that provides communication between the brain and a computer device. In other words it 'allows a person to communicate with or control the external world without using the brain's normal output pathways of peripheral nerves and muscles' [1], [2] [3]. In neuroscience experiments, the alternative output pathway for the brain can be one of the brain imaging methods, e.g., EEG, fMRI, and MEG. Two types of BCIs have been deployed in several studies. The first type is called Invasive in which the brain's neural activity is recorded by placing microelectrodes under a person's scalp (intracortical electrodes). A study conducted based on invasive brain imaging technique has shown the feasibility of planting some microelectrodes under a patient's scalp to generate a signal that could be used to move a cursor on the screen [4] (Kennedy et al., 2000). In spite of the fact that invasive methods record electrical activity with high-quality resolution, their use is rare due to the risk coming from the need to undertake neurosurgery to plant the electrodes.The second type is the non-invasive BCIs in which the brain's activity is recorded directly by using an appropriate brain imaging technique. It is obvious that non-invasive methods have some favourable aspects such as their simple use, durability, and the ability to use them without any risk for participants [5], [6].
BCIs that require the brain's output pathways (e.g., eye muscle) to stimulate the brain activity are called dependent BCIs. An example of this type of BCI is the one that presents a user with a matrix of characters and asks the user to spell a word by using eye gaze [7]. On the other hand, the independent BCI relies directly on the neural activity that is generated directly and immediately in response to a stimulus, such as a BCI that works by using only its user's attention to detect a target stimulus [8].
All BCIs that have been developed so far rely either on the visual or auditory input pathways (except one, which requires touch [9]). Displaying information to the user is generally considered to be quicker than playing audio, but sometimes audio is preferred (e.g. in case of a particular disability or in conjunction with, but not concurrent to, visual stimuli in order to reduce eye strain). In such cases, auditory stimuli can be used without any major concern, apart from in respect of speed [10].

A.
Stimuli presentation Different presentation techniques have been used in BCI. However, most of the paradigms were built using the Matrix design. The matrix paradigm (oddball) was first presented by [11] and is considered to be the most commonly used in brain computer interface (BCI) experiments [12]. The paradigm consists of presenting a participant with a matrix n × n of specific items (usually alphabet, digits, or alphanumeric characters). The rows and columns of the matrix are intensified in a pseudo-random sequence. Each flash lasts for a specific period (in milliseconds). A participant is usually presented with a word and the task is to spell that word by paying attention to the intensified character in the desired word and to mentally count how many times the row/column containing the target character is flashed (a standard design of matrix speller is shown in Fig. 1). Fig. 1 The participant usually sits in front of a screen at a specific distance (usually 60 cm). The 6 × 6 matrix is centred on the screen, The participant's task is to focus on the intended character in the desired word 'THE QUICK' (reintroduced from [7], [13]). The other paradigm utilises Rapid Serial Visual Presentation technique. RSVP is a presentation technique that allows a series of items to be presented in a specific position on a screen with a fixed rate. The presentation rate varies depending on the nature of the experiment, but commonly ranges between 6 to 20 items per second [14]. RSVP allows the presentation of a variety of items such as photos, words, letters, etc. Despite the feasibility of using RSVP as a presentation method, some perceptual errors may occur in RSVP-based experiments. Attentional blink (AB) is one of these errors that may occur during RSVP experiments. It can be observed when the SOA between two salient stimuli is less than 500 ms [14].

B.
BCI Structure Fig. 1 The standard design of EEG-based BCI. A subject's brain activity is recorded using the EEG system; the EEG data is then analysed in order to reveal some features, then the features are classified and translated to actions. C.
Neural Activity Acquisition Fig. 2 displays the main steps for designing a BCI. In an experiment, the brain's signals (BCI's inputs) are acquired from the participant's brain using one of the brain imaging techniques mentioned earlier. A BCI receives two types of inputs: evoked input or spontaneous input [15]. Evoked input can be elicited from the brain in response to stimulation (e.g., EEG signals produced by focusing on a target stimulus). Spontaneous inputs are not generated by stimulation; they reflect some aspects of the brain function during mental tasks [e.g., rhythmic activity [5]). Different brain imaging have been developed for several purposes such as EEG, fMRI and MEG. EEG is heavily used in BCI research due to its practical advantages: both the running and installation costs are low, portability makes it easier to conduct research and the high temporal resolution makes it ideal for fast-responding BCIs. BCIs were imagined as soon as EEG was discovered, but the computational speeds at the time were far from adequate. fMRI is also have been used in developing BCIs. Most of the fMRI-based BCI are in research on language, emotions, vision, psychological disorders and all research fields that employ neurofeedback and require high spatial resolution [16]. fMRI can tell which part of the brain is currently more active, and a patient using it can be informed of this in real time (with delays of a few seconds, which are not critical for this type of research). First, the raw fMRI output from a specific part of the brain is analysed. The system will then abstract from the resulting information to something more understandable, which is displayed back to the user in the scanner. The user then tries to manipulate his/her brain activity by looking at the feedback, closing the loop of interaction. For example, a user could manipulate his/her perception of pain by learning how to manipulate the region of the brain involved in pain perception [13].
BCIs based on MEG are currently being investigated, as it is believed that the higher spatial resolution of MEG, when compared to EEG, can increase the final speed of a BCI [17]. Even so, the practical drawbacks of MEG allow them to be used only in specific scenarios; therefore, like fMRI, they are used purely for research purposes.

D.
Neural Activity Processing Once the brain's signals are recorded, they are processed in order to improve the accuracy of the next step, which is the classification. In fact, the brain activity recorded by a brain imaging technique usually has a low signal-to-noise ratio (SNR). Thus, the noise removal (artefact rejection) is a crucial step in processing electrical data. While designing a BCI, the term 'noise' includes all data that are not relevant for BCI work. During an experiment, noise can be generated from different sources including either physiological (e.g., eye blink, or heartbeat) or environmental sources (e.g., system equipment, electrode status) [18]. Some types of noise can be detected easily even by inspection, such as an eye blink, which is usually generated on the frontal area of the scalp. These types of noise are usually removed from the recorded data by simply removing all segments that include them. However, in some occasions, the amount of noise can be vast, especially when the data are recorded from children or patients [19]. Thus, some approaches have been developed in order to detect and remove the amount of noise properly without any effects on the relevant data. Independent component analysis (ICA) is an advanced technique used commonly to treat noisy data systematically [20]. Feature selection is another step that follows artefacts rejection. Some prominent features can be extracted from the data. In fact, different BCIs can be designed based on the features that have been selected such as mu and beat rhythm activity [15], P300 evoked potentials [2], [3], [21], [22], slow cortical potentials [23] (Zhao et al., 2009), and SSVEPs component [24], [25].

E.
Feature Extraction and selection In order to develop a BCI, it is important to find a way to obtain the important characteristics of the brain signals. Different methods have been deployed and used to improve the BCI performance. This section outlines some feature extraction and selection techniques.

Principle component Analyses (PCA)
It is used to extract brain signal features by transforming a set of correlated variable into a smaller set of uncorrelated variable. In EEG-based BCI, PCA used to reduce the dimensions of the extracted features. It is also used for artefact reject in several BCI applications.

Indepaendent Component Analyses (ICA)
It is a statistical technique finds the sources of mixed signals without knowing any information about the characteristics of the signals. In BCI applications, ICA has been used to remove artefacts from noisy EEG signals [19]. It can be also deployed as a classification technique in some BCI studies [26].

F.
Data Classification Once the required features have been specified, algorithms are used to classify the features. Common techniques that have been used in developing BCIs include a non-linear algorithm; linear discriminant analysis; a support vector machine [27]; and algorithms based on neural networks. In fact, the choice of selecting the appropriate classification method depends on the nature of the BCI system that one aims to develop.

EXPERIMENT
In this paper, we analysed EEG data recorded from two BCIs paradigm designed by Srivas et al. The full details of the experiment can be found in [28]. A brief summary about the experiment procedure will be outlined in the following subsection.

G.
Participants The experiment included twelve participants, six female, six male.

H. Stimulus Presentation
The experiment consisted of two presentation modes: RSVP and Matrix.

I. RSVP Paradigm
The RSVP paradigm consisted of showing subjects six 5-letter words plus one word used for a training session. Each letter in these words was considered a target letter, so each word comprised five blocks (one per letter). Within each block, there were a number of successive RSVP runs ranging between 8 and 12. Each RSVP run consists of a stream of 25 uppercase English letters randomly chosen without repetition. The Stimulus Onset Asynchrony (SOA) was 166ms.

J.
Matrix Paradigm The Matrix paradigm (oddball) was first presented by Farwell and Donchin (1988) and is probably the most commonly used BCI speller. Similar to the RSVP paradigm, the approach consists of participants viewing six 5-letter words, and one further word used for training. Participants were faced with a 5 x 5 matrix, i.e. 25 uppercase English letters. The letters were presented in white color on a black background. For each word, participants viewed five blocks of a certain number of trials, ranging between 8 and 12. A run was defined as the flashing of all 5 rows and all 5 columns; thus, the probability of the target item being illuminated on the grid on each flash was 2/10, or 0.2. The rows and columns were illuminated in random order. Each flash lasted 133 ms, with an SOA of 166 ms.

K.
Experiment Tasks Each participant viewed both paradigms. Before initiating the stimulus presentation, participants were given IDs from 1 to 12. Odd-numbered participants performed the RSVP paradigm first, then Matrix, and vice versa for evennumbered participants. Prior to the start of the main experiment, each participant completed a training session including one word in each paradigm. Then, each participant was presented with six words in each paradigm: right, world, black, quest, flame, and heavy. The order of presentation of these words was randomly in both paradigms. For the duration of the presentation, participants were instructed to:  They were permitted to blink and relax during the questions and prior to initiating the next block.
 Mentally count how many times the target character was presented (in RSVP mode) or flashed (in Matrix mode).  Answer the counting question at the end of each block by choosing from multiple choices.

L.
Data analyse and classification EEG Data was filtered with a low-pass of 85Hz and a highpass of 0.30Hz. For further analyses (such as ERP generation), EEGLAB version 9 was used under Matlab 2010a. ERPs were segmented from -100ms to 800ms post critical stimuli, baseline corrected from -100ms to 0ms from critical item onset. In this paper, we investigate the performance of deep learning algorithms in in two different BCI aplications. Deep learning algorithms have been used in various BCI studies [29]- [31]. Two deepl learning algorithme will be used in this study: convolutional neural networks (CNN) and recurrent neural network (RNN). The block structure of any deep learning algorithm is present in Fig. 3.   Fig. 2 The main structure of Deep learning [32].  CNN CNN has been used heavily in EEG based applications. It has been applied successfully to classify SSVEPs [30]. In BCI, it was used detect P300 signal in P300 Speller application [33] [29]. Moreover, it provides a powerful tool for removing artefact for EEG signals [34]. The main characteristic of CNN is that it can extract the feature without using other feature extraction techniques. It consists of multi layers that are updated using error back propagation. In this paper, the following layers were optimized: convolutional layer, fully connected layer, pooling layer and the filter. Table-I lists the parameters used for optimizing the neural network. Fig. 4 presents the structure of CNN algorithm.

 RNN
RNN has been also used in BCI applications: automated visual classification [35] and smart sensor manufacturing environment [36]. RNN learns from the previous inputs, in other words its decision is affected by the past. Unlike the CNN, RNNs learn from the training as well as form the previous inputs in order to generate outputs. The RNN model used in this paper was similar to the one applied in [29]. The typical structure of RNN is blotted in Fig. 5.
 SVM The SVM is a statistical learning method used for analyzing data and classification patterns. It has been reported that the SVM performed better in EEG-based BCI The main idea of SVM is to construct a hyperplane (or multiple hyperplanes) in a high-dimensional space that classifies two or more classes. A high level of accuracy can be achieved when the distance between the hyperplane and the nearest data point of any class is large. The selection of training and testing data was determined by adopting 4-fold cross-validation. Fifth, the SVM was applied with two different kernels: a linear kernel and a Gaussian radial basis function (RBF) kernel. The two parameters of the RBF kernel (the kernel width and the cost parameter) were adjusted using 10-fold cross-validation. Finally, the svmclassify Matlab function was used to evaluate the classifier performance.

Fig. 4 The block structure of RNN model.  K-NN
The K-NN algorithm is a non-parametric technique in which the proximity is used to form the grouping [37]. The input is the K closest training cases among the feature sets. The k-NN was examined on different k values form 1 to 12. In order to measure the distance, the Euclidean distance was used.

M.
Classification approach of the P300 The aim was to discriminate between targets and non-targets based on the P300 of each single trial. In order to generate feature for the classification phase, Event Related Spectral Perturbation (ERSP) was applied to the raw EEG data. The ERSP calculates the mean changes in the power over different frequencies for single trials that are time locked to the same stimulus. The output of the ERSP is a 2D Matrix of log spectral difference from the baseline (in dB). EEGLAB software was used to calculate the ERSP [38], [39]. Fig.  A.1 plots the out of ERSP analyses for subject 2 EEG data. In order to compute the ERSP, the following formula was used: where t , f, and k refer to time point, frequency and trial. The number of all trials is defined b n. Each classifier was applied to a window from 300 to 800 ms from stimulus onset. This window was then sampled down to 100 epochs. The average of each segment was computed and considered as a feature. Thus, for each condition we have a set of N*100 data points, where N is the number of trilas. Ten-fold cross-validation was applied. The steps of EEG data classification are shown in Fig. 7. We have also used ROC curve analyses to get a better insight about the performance of each classifier. ROC curves of subject 7 in Matrix and RSVP are plotted in Fig. 7 and Fig.8 respectively.

RESULTS AND DISCUSSION
Results of the different algorithms applied to Matrix and RSVP data are presented in table-II and table-III respectively. As can be seen from the tables, CNN has outperformed all the other algorithms in both data sets. Linear SVM also showed high performance comparing to the other traditional algorithms. Interestingly, linear SVM has also produced higher accuracies comparing to the deep learning algorithm of RNN. Gaussian SVM has the worst performance when applied to the both data sets. It has been reported that the SVM performed better in EEG-based BCI The main idea of SVM is to construct a hyperplane (or multiple hyperplanes) in a high-dimensional space that classifies two or more classes. A high level of accuracy can be achieved when the distance between the hyperplane and the nearest data point of any class is large. In our experiment, The selection of training and testing data was determined by adopting 4-fold crossvalidation. Fifth, the SVM was applied with two different kernels: a linear kernel and a Gaussian radial basis function (RBF) kernel. The two parameters of the RBF kernel (the kernel width and the cost parameter) were adjusted using 10-fold cross-validation. Finally, the svmclassify Matlab function was used to evaluate the classifier performance. It is worth to note that Matrix data has higher accuracies with all the different algorithms when compared to the RSVP data. Our finding shows that Matrix-based BCI achieved higher performance in terms of detecting the P300. This finding comes in agreement with another study that compares between the performance of the two paradigms [28]. In [28], it has been shown that RSVP paradigm achieved less performance in terms of speller speed. Their study used Stepwise linear discriminant analyses to classify EEG data. The deep learning of CNN obtained the highest accuracies compared to the other machine learning algorithms. The linear SVM comes second in terms of achieving high accuracies. This result is also in line with [29]. The aim of their study was to exploare the visibility of applying deep learning algorithms in classifying EEG data of BCI applications. Their results approved the superiority of deep learning technique comparing to other machine learning algorithms.

CONCLUSION
Brain Computer Interface allows disabled people to communicate with the external world by using their brain signals. In this paper, the aim was to evaluate the performance of the traditional BCI paradigm which is based on the Matrix platform against the new paradigm that uses the RSVP technique. In Matrix, the paradigm consists of presenting a participant with a matrix n × n of specific items (usually alphabet, digits, or alphanumeric characters). On the other hand, RSVP paradigm consists of presenting a series of items in a specific position on a screen with a fixed rate. The aim of the paper was to apply deep learning algorithms in order to compare the performance of the Matrix-based BCI and RSVP-based BCI. Our finding shows that Matrixbased BCI achieved higher performance in terms of detecting the P300. This finding comes in agreement with another study that compares between the performance of the two paradigms [28]. In [28], it has been shown that RSVP paradigm achieved less performance in terms of speller speed. Their study used Stepwise linear discriminant analyses to classify EEG data. The deep learning of CNN obtained the highest accuracies compared to the other machine learning algorithms. The linear SVM comes second in terms of achieving high accuracies. This result is also in line with [29]. The aim of their study was to exploare the visibility of applying deep learning algorithms in classifying EEG data of BCI applications. Their results approved the superiority of deep learning technique comparing to other machine learning algorithms.
Despite the advantages of applying deep learning algorithms to EEG data, their performance is still not good enough to obtain higher accuracies. In our results, the highest accuracy was obtained in Matrix data with subject 4 (79.05%). This weak performance among all the algorithms suggests that the EEG data used in our paper contains noise, which might prevent the algorithms from detecting the P300 component. Another reason is due to limited number of EEG single trials used in our experiment. In the future, it is recommended to design a BCI with recording EEG data from several channels and over different locations at the scalp. This would provide enough EEG data sets that allows better training for machine learning algorithms.
APPENDIX I. Fig. A.1 presents the ERSP transform of Target and non-target trials for subject 2. The ERSP value at each time-frequency and time point is shown by a colour value. The red colour represents an increase of the power, while the green colour indicates no change in the power. As can be seen, the target trials show a power increase between 250ms to 500ms and over 0.5Hz to 10Hz frequencies. On the other hand, non-target trial produce no changes in the power during the sam time and frequencies.
The most right plot in the figure Depicts the ERSP of the difference between the ERSP of target trial and the ERSP of the non-target trials.