Hierarchical feature fusion framework for frequency recognition in SSVEP-based BCIs
Yangsong Zhang Erwei Yin Fali Li Yu Zhang Daqing Guo Dezhong Yao Peng Xu














mymainaddress,mythirdaddress,myfifthddress]Yangsong Zhang

mysecondaddress,mysixthaddress] Erwei Yinmycorrespondingauthor
[mycorrespondingauthor]Corresponding author
yinerwei1985@gmail.com

mythirdaddress,myfifthddress]Fali Li

myfourthaddress]Yu Zhang

mythirdaddress,myfifthddress]Daqing Guo



mythirdaddress,myfifthddress]Dezhong Yao

mythirdaddress,myfifthddress]Peng Xumycorrespondingauthor
xupeng@uestc.edu.cn

[mymainaddress]School of Computer Science and Technology, Southwest University of Science and Technology, Mianyang  621010, China
[mysecondaddress]National Institute of Defense Technology Innovation, Academy of Military Sciences China, Beijing, 100081 China
[mysixthaddress]Tianjin Artificial Intelligence Innovation Center (TAIIC), Tianjin, 300450 China
[mythirdaddress]Clinical Hospital of Chengdu Brain Science Institute, MOE Key Lab for Neuroinformation, University of Electronic Science and Technology of China, Chengdu 610054, China
[myfourthaddress]Department of Psychiatry and Behavior Sciences, Stanford University, Stanford, CA 94305, USA
[myfifthddress]School of life Science and technology, Center for information in medicine, University of Electronic Science and Technology of China, Chengdu, 611731, China


Effective frequency recognition algorithms are critical in steady-state visual evoked potential (SSVEP) based brain-computer interfaces (BCIs). In this study, we present a hierarchical feature fusion framework which can be used to design high-performance frequency recognition methods. The proposed framework includes two primary technique for fusing features: spatial dimension fusion (SD) and frequency dimension fusion (FD). Both SD and FD fusions are obtained using a weighted strategy with a nonlinear function. To assess our novel methods, we used the correlated component analysis (CORRCA) method to investigate the efficiency and effectiveness of the proposed framework. Experimental results were obtained from a benchmark dataset of thirty-five subjects and indicate that the extended CORRCA method used within the framework significantly outperforms the original CORCCA method. Accordingly, the proposed framework holds promise to enhance the performance of frequency recognition methods in SSVEP-based BCIs.




Brain-computer interface(BCI)feature fusionsteady-state visual evoked potential (SSVEP)correlated component analysis (CORRCA)electroencephalogram (EEG).
[2010] 00-0199-00








§ INTRODUCTION

A brain-computer interface (BCI) is a type of communication system that can directly translate brain signals into digital commands for the control of external devices without the involvement of the peripheral nerves and muscles. BCI systems show great promise for providing communication access to both people with severe motor disabilities and typically developed individuals alike  <cit.>. Due to its relative portability and low cost, electroencephalography (EEG) remains the most widely investigated sensing modality in BCI research due in part to its excellent temporal resolution <cit.>. To date, several types of EEG signals have been used to design and operate BCIs, e.g. ERP <cit.>, sensorimotor rhythm (SMR) <cit.>, steady-state visual evoked potential (SSVEP) <cit.>, and motion-onset visual evoked potentials <cit.>, etc. In addition, some researchers have begun to explore hybrid BCI approaches using multiple signals and modalities <cit.>. In recent years, an increasing number of researchers have focused on SSVEP-based BCIs because of their high information transfer rate (ITR) and minimal user training requirements  <cit.>.

For SSVEP-based BCIs, an effective frequency detection algorithm plays an important role in overall system performance  <cit.>. In literature, various techniques for SSVEP feature extraction and classification have been developed <cit.>. Among them, the most popular are based on multivariate statistical algorithms, such as canonical correlation analysis (CCA) <cit.>, and multivariate synchronization index (MSI) <cit.>, etc, which are easy to implement without complex optimization procedures. Recently, we proposed a novel frequency recognition method, termed CORRCA <cit.>, based on the correlated component analysis (COCA) which is also a multivariate statistical algorithm <cit.>. The CORRCA method significantly outperforms the state-of-art CCA method <cit.>. As we know, CCA is a multivariate statistical method that measures the correlation between two sets of signals <cit.>. It requires that the canonical projection vectors be orthogonal and it generates two different projection vectors for the two signals. In contrast, COCA aims to maximize the Pearson Product Moment Correlation coefficient and does not necessitate orthogonality between the projection vectors. Furthermore, it generates only one projection vectors for the two signals this simplifying subsequent analysis <cit.>. For EEG signal analysis, the COCA method maybe more efficient and practical in real-world applications <cit.>.

Although standard CCA and CORRCA algorithms have been applied in literature with satisfactory results, the performance of these system could be further improved by applying sophisticated signal processing technologies. One such approach is the filter bank analysis method <cit.>, which employs several bandpass filters to generate multiple subband components of the input signals. This filter bank based approach first carries out a frequency recognition method to obtain features on each subband component, then integrates all the features in classification using a weighted combination. In fact, we can consider the filter bank technique as a form feature fusion in frequency domain. This fusion strategy is just one way to explore the discriminating information implicit in the original signals. Here we term it a frequency domain fusion (FD fusion). In a recent motor imagery based BCI study, it was determined that weighting and regularizing common spatial patterns (CSP) features allow for the use of preclude the need for feature selection thus avoiding the loss of valuable information, and enhancing the performance of CSP <cit.>. We can consider this type of feature weighted method as a feature fusion strategy in the spatial domain. Analogously, we term it the spatial domain fusion(SD fusion). Both the standard CCA method and CORRCA method provide multiple correlation coefficients to measure the correlation between two multi-dimensional signals. It is worth noting that traditionally only the largest coefficients is selected as feature while all others are ignored <cit.>. This inevitably results in the loss of the discriminative information. We posit that merging these two fusion strategies will result in a more robust target detection framework for SSVEP-based BCIs. To the best of our knowledge, a framework including both SD and FD fusions has never been introduced in SSVEP-based BCI to date.

Motivated by previous studies, here we propose a hierarchical feature fusion framework for SSVEP target detection by implementing both SD and FD fusions. In order to evaluate the efficiency and effectiveness of the proposed framework, we utilized the CORRCA method as a representative frequency recognition algorithm to investigate if the proposed framework can affect its performance. Furthermore, we conducted extensive experimental evaluation on a benchmark dataset of thirty-five subjects.

The remainder of this paper is organized as follows. Section II describes the proposed framework as well as its implementation and application techniques; Section III describes the dataset and performance evaluation; Section IV presents the experimental results; and the last two sections discuss and conclude this study.




§ METHODS




 §.§ The hierarchical feature fusion framework and its implementation

The hierarchical feature fusion framework includes three main stages, i.e., bandpass filtering, hierarchical feature fusion and frequency recognition (see Fig. <ref>).

	
  * In the stage of bandpass filtering, several bandpass filters are employed to filter the EEG signals, respectively, into multiple sub-band components. The filter bank design for the bandpass filters should be optimized according to the frequencies of the SSVEP stimulus. Here, all the sub-bands cover multiple harmonic frequency bands with the same high cut-off frequency at the upper bound frequency. for a more detailed analysis for the frequency band selection refer to  <cit.>.
	
  * In the second stage, the hierarchical feature fusion includes SD fusion and FD fusion, respectively. First, for each pair of sub-band signals, a spatial filtering method yields multiple features that measure the correlation levels. Then, SD fusion is implemented on these features to obtain a new feature for each sub-band. Subsequently, FD fusion is used to generate the final features at each stimulus by combining the new features obtained by SD fusion on all the sub-bands. For both SD and FD fusions, many strategies could be used to fuse the features. In the present study, we used a weighted method with a nonlinear function to combine all the features obtained via the spatial filtering method. For the FD fusion method, we also implemented a weighted strategy with nonlinear function to combine all the features.
	
  * In the third stage, the proposed framework performs the frequency recognition based on the features obtained in the second stage. Supervised learning methods, such as linear discriminant analysis (LDA), deep learning, as well as unsupervised learning methods can be used in this stage. Here, we employ the latter because of its satisfactory performance in previous studies <cit.>. Specifically, the target frequency can be recognized as that with the highest magnitude of final feature.





In the implementation of the framework, no prior knowledge about the weights can be directly inferred in the SD fusion step, and the weights themselves might depend on the chosen spatial filtering method. In this subsection, these weights are denoted as ϕ_1,ϕ_2, ⋯,ϕ_C.

Denote a test sample as Z∈R^C × N and the i-th frequency template signal as Y^i∈R^C × N (i=1,2,⋯,N_f). Furthermore, denote the sub-band signals of Z and Y^i  after the l-th bandpass filtering as Z_l∈R^C × N and Y^i_l∈R^C × N respectively, l=1,2,⋯,SN;    i=1,2,⋯,N_f. With a spatial filtering method, C features in descending order, denoted as [ λ^l_1,   λ^l_2,   ⋯  ,  λ^l_C ], on the Z_l and Y^i_l  can be obtained. Here, C is the number of variables, N is the number of samples, N_f is the number of stimulus frequencies, and SN is the number of bandpass filters.

Then with the SD fusion, the new feature in l-th sub-and at the i-th frequency can be obtained as :


    δ^i_l= ∑_k=1^Cϕ_k·λ^l_k,    l=1,2,⋯,SN,    i=1,2,⋯,N_f.


With FD fusion, the final feature for the test sample Z and the template signal Y^i at the i-th frequency (i=1,2,⋯,N_f) can be obtained as :


    ψ_i= ∑_l=1^SN w_l·δ^i_l,    i=1,2,⋯,N_f.

where w_l,w_2,⋯,w_SN are the weights. They can be obtained by a weighting nonlinear function as below (refer to  <cit.>):


    w_m= m^-a_1+b_1    ,m=1,2,⋯,SN.

where a_1 and b_1 are two parameters that control the classification performance, respectively.

Then, the frequency f_Z of  Z is specified as the frequency of the template signal that has maximal feature with Z, as below:
 
    f_Z=max_f ψ_i(f)  ,    i= 1, 2,…,N_f




 §.§ The application of the framework on standard CORRCA method

CORRCA method is developed based on the COCA, which is a technique to maximize the Pearson Product Moment Correlation coefficient between two multi-dimensional signals <cit.>. Compared to CCA, COCA relaxes the constraint on orthogonality among the projection vectors, and generates a single projection vector for the two multi-dimensional input signals. Mathematically, COCA is an optimization problem, and its projection vectors can be obtained by solving a generalized eigenvalue problem. COCA has been used to investigate cross-subject synchrony of neural processing <cit.>, and inter-subject correlation in evoked encephalographic responses <cit.>. Recently, it was introduced for frequency recognition in SSVEP-based BCIs <cit.>. Below we provide a brief description of the standard CORRCA method.

Denote X∈R^C × N and  Y∈R^C × N as two multidimensional variables, where C is the number of variables and N the number of samples. COCA  seeks to find a projection vector w∈ℝ^C × 1 such that the resulting linear combinations  x=w^TX and y=w^TY  exhibit maximal correlation.


    ρ   =max_wxy^T/x y
       =max_ww^TR_12w/√(w^TR_11w)√(w^TR_22w)

where ρ denotes the correlation coefficient. R_11=1/NXX^T, R_22=1/NYY^T, R_12=1/NXY^T, and R_21=1/NYX^T are four sample covariance matrices. Supposing that w^TR_11w = w^TR_22w, differentiating formula (<ref>) with respect to w and setting to zero, we obtain the following eigenvalue equation <cit.>:


    (R_12+R_21) w =λ (R_11 + R_22) w


With formula (<ref>), we can obtain C correlation coefficients,  i.e., ρ_1,ρ_2,⋯,ρ_C for X and Y. In the standard CORRCA method, only the maximal coefficient among {ρ_1,ρ_2,⋯,ρ_C}  is used as the  feature for  frequency recognition.

Suppose that Z̅∈R^C × N is a test sample, Y̅_i∈R^C × N is an individual template signal calculated by averaging SSVEP data across multiple trials at frequency f_i, i=1,2,⋯,N_f. Denote maximal correlation coefficients between Z̅ and Y̅_i as β_i, i=1,2,⋯,N_f. Then, the frequency of  Z̅ is determined by finding the frequency of the template signal that has maximal correlation with Z̅, as below:
 
    f_Z̅=max_f β_i(f)  ,    i= 1, 2,…,N_f




Although the CORRCA method yields significantly better performance compared to the state-of-art CCA method, only the largest correlation coefficients in the set of {ρ_1,ρ_2,⋯,ρ_C}  is retained and the others are optionally discarded. In our preliminary analysis on the benchmark dataset, we investigated the classification performance of each correlation coefficient in the CORRCA method. As shown in Fig.<ref>,  the correlation coefficient index versus the averaged classification accuracy across subjects at four time windows is presented. Notice that almost all the coefficients could provide discriminative information for achieving classification accuracies above chance level (1/40). Therefore, exploring the strategy of combining all the correlation coefficients, namely through SD fusion, could be beneficial for enhancing the overall performance of the CORRCA method. In addition, the previous study demonstrated that the FD fusion can improve the performance of CORRCA method <cit.>. Accordingly, we believe that the proposed framework could further enhance system performance compared to using either SD fusion or FD fusion alone. In the current study, the CORRCA method was used as a representative method to implement the proposed framework, and investigate how its performance is affected.

In implementing the framework on the CORRCA method, no prior knowledge about the weights in the SD fusion step can be directly inferred.  As such, the key question optimal design of feature weights, i.e., the correlation coefficients. Intuitively, the discriminative ability should vary among the features, and the accuracy curve obtained with differing relative feature weights should be nonlinear. As shown in Fig.<ref>,  we find that the system accuracy decreases nonlinearly from the largest correlation coefficient to the smallest one, and generally the larger coefficients yield higher accuracies. The accuracy curve most closely follows an exponential function (as seen in Fig.<ref>). Accordingly, we adopted a weighting nonlinear function to calculate the weights in the SD fusion, as below.


    ϕ_k=  e^(-a_2 · k)+b_2    ,k=1,2,⋯,C.

where a_2 and b_2 are two parameters that control the classification performance. In current study, a_2 and b_2 were optimized via a grid search using a standard CORRCA method on the training set. Fig. 3 depicts the parameter selection process in which a_2 and b_2 were limited to [0:0.2:2] and [0:0.2:2], respectively. Here, the weights were calculated using formula (<ref>) with a_0 and b_0 specified as default values of 1.25 and 0.25, respectively. The classification accuracy and ITR with different a_2 and b_2 values with a time window of 0.8 s were calculated. The parameter values of a_2 and b_2 that led to the highest ITR were selected (i.e. 0.6 and 0), respectively. Then these values were applited onto the test set in the following calculation and analysis. More details of the corresponding procedure and parameter settings employed for bandpass filtering can be found in our previous study <cit.>. Note that the choice of formula (<ref>) is not unique, other function based may also be adequate. The CORRCA method with the proposed framework is termed as the hierarchical feature fusion CORRCA (HFCORRCA) hereafter.






§ EXPERIMENTAL STUDY



 §.§ EEG dataset

The EEG dataset used for evaluation is a publicly available benchmark dataset consisting of offline SSVEP-based BCI spelling experiments on thirty-five healthy subjects (seventeen females, mean age 22 years) <cit.>. In the BCI system, 40 frequencies (8-15.8 Hz with an interval of 0.2 Hz) were used for target coding. During the experiments, each participant was asked to spell for six blocks. In each block, 40 trials corresponding to all 40 stimulus frequencies were provided in a random order. Each trial lasted for 6 s, including 0.5 s for the visual cue and 0.5 s for the inter-stimulus interval.

EEG data were recorded at a sampling rate of 1000 Hz via a Synamps2 system (Neuroscan, Inc.). A total of 64 electrodes were placed according to the 64-channel extended international 10-20 standard. The EEG data were treated with a 50-Hz notch filter and subsequently with a 0.15-200-Hz band-pass filter. Then, each 6-s segment beginning at 0.5 s pre-stimulus was extracted. After that, the epoch was downsampled to 250 Hz. A more detailed description of the EEG dataset can be found in the literature <cit.>.

In current study, we divided the dataset into two parts, i.e. a training set and testing set. The data from first fifteen subjects were used as training set, and the remaining twenty subjects were used as the testing set.



 §.§ Performance evaluation

In this study, we carry out an extensive comparison between the spatial filtering method with and without the proposed framework. For the spatial filtering method, CORRCA was used because of its robust performance. The classification accuracy and ITR were adopted as the evaluation metric. The 0.5 s cue time was considered in the calculation of ITR. In the current study, the individual templates of each frequency were generated by averaging SSVEP data across multiple blocks at the corresponding frequency. We adopted the leave-one-block-out cross validation method to evaluate the performance for the comparisons between the two methods. Specifically, one EEG block from a group of five blocks was selected as the training set and the remaining blocks were treated as the testing set. For each subject, this procedure was repeated six times such that the samples of each block were used as the testing set once.

Furthermore, we used the r-square value to evaluate the discriminability of the features obtained by each method. In the current study, the r-square value was computed using feature values of the attended target stimulus and the maximal feature values of the non-attended stimuli <cit.>, as the following formula <cit.>.


    R =√(N_1 · N_2)/N_1+N_2·mean(F_1)-mean(F_2)/std(F_1∪ F_2)

where F_1 and F_2 are the sets of features of attended target stimulus and the maximal feature values of the non-attended stimuli, respectively. N_1 and N_2
are the numbers of features in these two sets. As the sign of this difference is important, we adopt the signed r-square value calculated as follows <cit.>:


    r^2 = Sign(R ) · R^2




§ RESULTS

Fig.<ref> shows classification accuracies and ITRs averaged across all subjects at different time windows ranging from 0.2 s to 1 s with a step size of 0.1 s. As shown in Fig.<ref>, the HFCORRCA method consistently outperforms the original CORRCA method in terms of accuracy and ITR, and their difference is significant when the time windows are above 0.2 s (paired t-tests). These results indicate that the proposed framework is feasible and can enhance the performance of the CORRCA method.



To further compare the performance between CORRCA and HFCORRCA methods, we present the averaged classification accuracies and the signed r-square values across all subjects at the 40 stimulus frequencies with a time window size of 1 s. As shown in Fig.<ref>, the HFCORRCA method increased the vast majority of classification accuracies (37/40) and the r-square values (32/40) of the forty stimulus frequencies. Compared to the CORRCA, the standard deviation of classification accuracies using the HFCORRCA method decreased from 7.73 to 3.55, which indicates that HFCORRCA improves the reliability of frequency detection for all frequencies. In addition, the mean value of signed r-square values averaged across the forty frequencies increased from 0.39 to 0.48. These results further demonstrate that the proposed framework can produce more robust features compared to existing methods, and improve of the discriminability between attended target stimulus and non-attended non-target stimuli.






Previous studies have demonstrated that the number of channels and training trials can significantly influence performance <cit.>. Here we further investigate the classification accuracies of HFCORRCA and CORRCA methods at different number of channels and training trials. Fig.<ref> illustrates the results of both methods using a 1-s time window. As shown, the HFCORRCA method yielded significantly better performance than the CORRCA method under all conditions. These results demonstrate that the proposed framework is a promising strategy to improve the performance of frequency recognition methods, such as the CORRCA method, to further enhance the performance of real-world systems.



§ DISCUSSION

EEG is the most studied modality in BCI research, however, it is highly prone to be contamination by noise and artifacts. These factors usually distort the useful components in the signal and result in dispersion of the discriminative information across the projected signals or features <cit.>. Mishuhina et al. found that it is beneficial in motor imagery data processing to combine features obtained using CSP spatial filters <cit.>. Based on this prior research, we assume that fusing the features produced by different frequency recognition spatial filters can also provide more robust features and avoiding loss of the discriminative information for frequency recognition in SSVEP-based BCIs (SD fusion). For instance, the CORRCA method utilizes only the largest correlation coefficient corresponding to the spatial filter of the largest generalized eigenvalues, and all other features are discarded. Fusing all the correlation coefficients generated by all the spatial filters of CORRCA method can enhance overall performance. Additionally, filter bank methods are widely used in SSVEP-based BCI and motor imagery based BCIs to generate more discriminative features <cit.>. This technique achieves feature fusion in different frequency bands (FD fusion). Traditionally, these two types of features fusion methods are used independently in the BCI studies. To further boost the performance of SSVEP-based BCI systems, we propose a unified framework to integrate SD and FD fusion together for frequency recognition. The experimental results indicate that the framework can boost the frequency recognition when used with existing method, e.g., the CORRCA method, to enhance its performance. Specifically, the CORRCA method within the proposed framework significantly outperforms the contemporary methods under various verification conditions. Based on the proposed framework, some extended methods of spatial filtering are simply special cases of the proposed method. For instance,  filter bank CORRCA method is the HFCORRCA method without SD fusion.



The standard CORRCA method can be considered within out framework by applying only the SD fusion on CORRCA method with a step function that has sharp transition from 1 to 0. This result of this is that only the largest coefficient is choosen. Conversely, for the HFCORRCA, we adopted a soft weighted function as formula (<ref>) to combine all the correlation coefficients. We kept all coefficients, and combined them with different weights. As shown Fig.<ref>, we found that the accuracies decreased nonlinearly from the largest correlation coefficient to the smallest one. Intuitively, it is reasonable to use different weights on these coefficients. To verify this idea, as shown in Fig.<ref>, we present the results obtained by the CORRCA, eHFCORRCA (a method that replaces the different weights used in HFCORRCA to equal weights, i.e., ϕ_1=1, ϕ_2=1, ⋯, ϕ_C=1) and HFCORCA.  Although eHFCORRCA shows better performance than CORRCA at short time windows (less than 0.5 s), we find that using equal weights produces inferior results during fusion of features using the CORRCA method. These results further confirm the justification for adopting a nonlinear weighted  function to fuse the features. Note that the weighted function in formula (<ref>) can be changed to other functions, such as w_k=k^-a+b (k=1,2,⋯,C) in formula (<ref>). In future studies, we will investigate different weighting functions for different spatial filtering methods, which might further optimize system performance.


To confirm the contribution of each fusion operation (SD fusion and FD fusion) of the proposed framework, we calculated the classification accuracies using the CORRCA method with SD fusion only and FD fusion(FFCORRCA) only. As shown in Fig.<ref>, although both SD fusion and FD fusion can improve the results compared to standard CORRCA, respectively, combining the two fusion methods resulted in the best performance (except at the shortest time window of 0.2 s). At the time window of 0.2 s, the CORRCA method with FD fusion produced lower accuracies than the original CORRCA method, which may be attributed to the short data length (50 samples) during bandpass filtering. It is believed that SD fusion in general is less sensitive to data length than FD fusion.





 In applying the fusion methods, we fused all the features in SD fusion step, and used five bandpass filters based on previous study <cit.>. We did not optimize the two parameters of C and SN. As shown in Fig.<ref>, the accuracies of the last four correlation coefficients are much lower than those of the first three correlation coefficients. One may argue that it is unnecessary to use all the correlation coefficients in the proposed framework. In order to provide some guidance for future online system development using HFCORRCA, we investigated how these two parameters influence the classification accuracies.  The C and SN were limited to [2:9] and [1:5], respectively, and two data time windows, i.e., 0.8 s, 1 s were used. The upper limit of SN is equal to the number of channels in the EEG. As shown in Fig.<ref>, for the two chosen time windows, the values of SN which yielded the best results are same, but those of C are different. We assessed the system accuracies when C ranged from 4 to 9, and found that the accuracy differed by less than 0.5% across the range. That is to say, the first four correlation coefficients can yield satisfactory results on the benchmark dataset. The optimal value of C and SN may depend on the specific dataset. For SN, when the stimulus frequencies belong to the same range used as those used in the benchmark dataset, it can be set to 3. For C, the simplest way is to set it equal to the number of the features produced by the SF method. When a calibration dataset is available, preliminary experiments can be used to optimize these values.



Lastly, it is worth mentioning that the proposed framework can only be applied directly on the standard frequency recognition methods (e.g., CCA, CORRCA) for now. In a future study, we will explore integrating this framework into extended frequency recognition methods, such as the two-stage method based on the standard CORRCA method (TSCORRCA) <cit.>. Moreover, the proposed framework can also be employed on CSP methods to boost classification rates in motor imagery based BCIs <cit.>.



§ CONCLUSION

In summary, we propose a hierarchical features fusion framework to address the loss of discriminative information the occurs during feature extraction of spatial filtering methods. The framework consists of feature fusion using a nonlinear weighted function in both spatial and frequency domains. Under this framework, more robust features for frequency recognition are generated. Experimental results performed on a benchmark dataset demonstrated that the framework is effective at enhancing system performance. Specifically, and the improved CORRCA method within this framework significantly outperforms the original CORRCA method. This novel framework may be used to develop efficient methods for frequency recognition in SSVEP-based BCIs.




§ ACKNOWLEDGMENTS

This work was supported in part by the National Natural Science Foundation of China under Grant 61871423, Grant 61703407, Grant 81401484,  Grant 31771149, Grant 61522105, Grant 61527815, and Grant 61773129, in part by the Longshan academic talent research supporting program of SWUST under Grant 17LZX692. We also acknowledge Mr. Rami Saab for editing an early version of this manuscript.



§ REFERENCES



10
url<#>1urlprefixURL href#1#2#2 #1#1

wolpaw2002
J. R. Wolpaw, N. Birbaumer, D. J. McFarland, G. Pfurtscheller, T. M. Vaughan,
  Brain-computer interfaces for communication and control, Clinical
  neurophysiology 113 (6) (2002) 767–791.

chaudhary2016brain
U. Chaudhary, N. Birbaumer, A. Ramos-Murguialday, Brain-computer interfaces for
  communication and rehabilitation, Nature Reviews Neurology 12 (9) (2016)
  513–525.

MBCILiyq2016
Y. Li, J. Pan, J. Long, T. Yu, F. Wang, Z. Yu, W. Wu, Multimodal BCIs: Target
  detection, multidimensional control, and awareness evaluation in patients
  with disorder of consciousness, Proc. IEEE 104 (2) (2016) 332–352.

chen2015PNAS
X. Chen, Y. Wang, M. Nakanishi, X. Gao, T.-P. Jung, S. Gao, High-speed spelling
  with a noninvasive brain-computer interface, Proc.Natl.Acad. Sci. USA.
  112 (44) (2015) E6058–E6067.

JingJ2011
J. Jin, B. Z. Allison, E. W. Sellers, C. Brunner, P. Horki, X. Wang, C. Neuper,
  An adaptive P300-based control system, J. Neural Eng. 8 (3) (2011) 036006.

townsend2016
G. Townsend, V. Platsko, Pushing the P300-based brain–computer interface
  beyond 100 bpm: Extending performance guided constraints into the temporal
  domain, J. Neural Eng. 13 (1) (2016) 026024.

xu2018
M. Xu, X. Xiao, Y. Wang, H. Qi, T.-P. Jung, D. Ming, A brain computer interface
  based on miniature event-related potentials induced by very small lateral
  visual stimuli, IEEE Trans. Biomed. Eng. 65 (5) (2018) 1166–1175.

feng2018
J. Feng, E. Yin, J. Jin, R. Saab, I. Daly, X. Wang, D. Hu, A. Cichocki, Towards
  correlation-based time window selection method for motor imagery BCIs,
  Neural Networks 102 (2018) 87–95.

Jiao2018
Y. Jiao, Y. Zhang, X. Chen, E. Yin, J. Jin, X. y. Wang, A. Cichocki, Sparse
  group representation model for motor imagery eeg classification, IEEE J
  Biomed Health Inf. (2018) 1–1http://dx.doi.org/10.1109/JBHI.2018.2832538
  doi:10.1109/JBHI.2018.2832538.

liuyy2018
Y. Yu, Y. Liu, J. Jiang, E. Yin, Z. Zhou, D. Hu, An asynchronous control
  paradigm based on sequential motor imagery and its application in wheelchair
  navigation, IEEE Trans. Neural Syst. Rehabil. Eng. 26 (12) (2018) 2367–2375.

MFSC2012
Y. Zhang, P. Xu, T. Liu, J. Hu, R. Zhang, D. Yao, Multiple frequencies
  sequential coding for SSVEP-based brain-computer interface, PLoS ONE 7 (3)
  (2012) e29519.

Maye2017
A. Maye, D. Zhang, A. K. Engel, Utilizing retinotopic mapping for a
  multi-target SSVEP BCI with a single flicker frequency, IEEE Trans.
  Neural Syst. Rehabil. Eng. 25 (7) (2017) 1026–1036.

Oikonomou2018
V. P. Oikonomou, S. Nikolopoulos, I. Kompatsiaris, A bayesian multiple kernel
  learning algorithm for ssvep bci detection, IEEE J Biomed Health Inf. (2018)
  1–1http://dx.doi.org/10.1109/JBHI.2018.2878048
  doi:10.1109/JBHI.2018.2878048.

guo2008
F. Guo, B. Hong, X. Gao, S. Gao, A brain-computer interface using motion-onset
  visual evoked potential, J. Neural Eng. 5 (4) (2008) 477.

ma2017
T. Ma, H. Li, H. Yang, X. Lv, P. Li, T. Liu, D. Yao, P. Xu, The extraction of
  motion-onset vep bci features based on deep learning and compressed sensing,
  J. Neurosci. Methods 275 (2017) 80–92.

liyq2010
Y. Li, J. Long, T. Yu, Z. Yu, C. Wang, H. Zhang, C. Guan, An EEG-based BCI
  system for 2-D cursor control by combining mu/beta rhythm and P300
  potential, IEEE Trans. Biomed. Eng. 57 (10) (2010) 2495–2505.

yin2015hybrid
E. Yin, T. Zeyl, R. Saab, T. Chau, D. Hu, Z. Zhou, A hybrid brain-computer
  interface based on the fusion of P300 and SSVEP scores, IEEE Trans.
  Neural Syst. Rehabil. Eng. 23 (4) (2015) 693–701.

xu2013
M. Xu, H. Qi, B. Wan, T. Yin, Z. Liu, D. Ming, A hybrid bci speller paradigm
  combining p300 potential and the ssvep blocking feature, J. Neural Eng.
  10 (2) (2013) 026001.

Edelman2018
B. J. Edelman, J. Meng, N. Gulachek, C. C. Cline, B. He, Exploring cognitive
  flexibility with a noninvasive BCI using simultaneous steady-state visual
  evoked potentials and sensorimotor rhythms, IEEE Trans. Neural Syst. Rehabil.
  Eng. 26 (5) (2018) 936–947.
http://dx.doi.org/10.1109/TNSRE.2018.2817924
  doi:10.1109/TNSRE.2018.2817924.

Yin2015
E. Yin, Z. Zhou, J. Jiang, Y. Yu, D. Hu, A dynamically optimized SSVEP
  brain-computer interface (BCI) speller, IEEE Trans. Biomed. Eng. 62 (6)
  (2015) 1447–1456.

chen2014
X. Chen, Z. Chen, S. Gao, X. Gao, A high-ITR ssvep-based BCI speller,
  Brain-Comp.Interfaces 1 (3-4) (2014) 181–191.

Jiao2017
Y. Jiao, Y. Zhang, Y. Wang, B. Wang, J. Jin, X. Wang, A novel multilayer
  correlation maximization model for improving CCA-based frequency
  recognition in SSVEP brain-computer interface, Int. J. Neural Syst. 0 (0)
  (2017) 1750039.
http://dx.doi.org/10.1142/S0129065717500393
  doi:10.1142/S0129065717500393.

zhang2018a
Y. Zhang, D. Guo, F. Li, E. Yin, Y. Zhang, P. Li, Q. Zhao, T. Tanaka, D. Yao,
  P. Xu, Correlated component analysis for enhancing the performance of
  SSVEP-based brain-computer interface., IEEE Trans. Neural Syst. Rehabil.
  Eng. 26 (5) (2018) 948–956.

jiang2018
J. Jiang, E. Yin, C. Wang, M. Xu, D. Ming, Incorporation of dynamic stopping
  strategy into the high-speed SSVEP-based BCIs, J. Neural Eng. 15 (4)
  (2018) 046025.

friman2007
O. Friman, I. Volosyak, A. Graser, Multiple channel detection of steady-state
  visual evoked potentials for brain-computer interfaces, IEEE Trans. Biomed.
  Eng. 54 (4) (2007) 742–750.

Zerafa2018
R. Zerafa, T. Camilleri, O. Falzon, K. P. Camilleri, To train or not to train?
  a survey on training of feature extraction methods for SSVEP-based BCIs,
  J. Neural Eng. 15 (5) (2018) 051001.

Lin2007
Z. Lin, C. Zhang, W. Wu, X. Gao, Frequency recognition based on canonical
  correlation analysis for SSVEP-based BCIs, IEEE Trans. Biomed. Eng.
  54 (6) (2007) 1172–1176.

chen2015filter
X. Chen, Y. Wang, S. Gao, T.-P. Jung, X. Gao, Filter bank canonical correlation
  analysis for implementing a high-speed SSVEP-based brain-computer
  interface, J. Neural Eng. 12 (4) (2015) 046008.

MSI2014
Y. Zhang, P. Xu, K. Cheng, D. Yao, Multivariate synchronization index for
  frequency recognition of SSVEP-based brain-computer interface, J. Neurosci.
  Methods 221 (2014) 32–40.

ZhangRMSI2016
Y. Zhang, D. Guo, P. Xu, Y. Zhang, D. Yao, Robust frequency recognition for
  SSVEP-based BCI with temporally local multivariate synchronization index,
  Cognitive Neurodynamics 10 (6) (2016) 505–511.

zhangys2018b
Y. Zhang, E. Yin, F. Li, Y. Zhang, T. Tanaka, Q. Zhao, Y. Cui, P. Xu, D. Yao,
  D. Guo, Two-stage frequency recognition method based on correlated component
  analysis for SSVEP-based BCI, IEEE Trans. Neural Syst. Rehabil. Eng.
  26 (7) (2018) 1314–1323.

Dmochowski2012
J. Dmochowski, P. Sajda, J. Dias, L. Parra, Correlated components of ongoing
  EEG point to emotionally laden attention-a possible marker of engagement,
  Frontiers Hum. Neurosci. 6 (2012) 112.
http://dx.doi.org/10.3389/fnhum.2012.00112
  doi:10.3389/fnhum.2012.00112.

dmochowski2014audience
J. P. Dmochowski, M. A. Bezdek, B. P. Abelson, J. S. Johnson, E. H. Schumacher,
  L. C. Parra, Audience preferences are predicted by temporal reliability of
  neural processing, Nature Commun. 5 (2014) 4567.

CohenENEURO2016
S. S. Cohen, L. C. Parra, Memorable audiovisual narratives synchronize sensory
  and supramodal neural responses, eNeuro 3 (6).
http://dx.doi.org/10.1523/ENEURO.0203-16.2016
  doi:10.1523/ENEURO.0203-16.2016.

Hotelling1936
H. Hotelling, Relations between two sets of variates, Biometrika 28 (3/4)
  (1936) 321–377.

Pearson1896
K. Pearson, Vii. mathematical contributions to the theory of
  evolution.—iii. regression, heredity, and panmixia, Philos.
  Trans. R. Soc. Lond. A 187 (1896) 253–318.
http://dx.doi.org/10.1098/rsta.1896.0007
  doi:10.1098/rsta.1896.0007.

mishuhina2018
V. Mishuhina, X. Jiang, Feature weighting and regularization of common spatial
  patterns in EEG-based motor imagery BCI, IEEE Signal Processing Letters
  25 (6) (2018) 783–787.

nakanishi2015
M. Nakanishi, Y. Wang, Y.-T. Wang, T.-P. Jung, A comparison study of canonical
  correlation analysis based methods for detecting steady-state visual evoked
  potentials, PLoS ONE 10 (10) (2015) e0140703.

Ki2016
J. J. Ki, S. P. Kelly, L. C. Parra, Attention strongly modulates reliability of
  neural responses to naturalistic narrative stimuli, Journal of Neuroscience
  36 (10) (2016) 3092–3101.
http://dx.doi.org/10.1523/JNEUROSCI.2942-15.2016
  doi:10.1523/JNEUROSCI.2942-15.2016.

wangyj2016
Y. Wang, X. Chen, X. Gao, S. Gao, A benchmark dataset for SSVEP-based
  brain-computer interfaces, IEEE Trans. Neural Syst. Rehabil.Eng. 25 (10)
  (2017) 1746–1752.

blankertz2007
B. Blankertz, G. Dornhege, M. Krauledat, K.-R. Müller, G. Curio, The
  non-invasive Berlin brain-computer interface: fast acquisition of effective
  performance in untrained subjects, NeuroImage 37 (2) (2007) 539–550.

Zhangyu2017NS
Y. Zhang, Y. Wang, J. Jin, X. Wang, Sparse bayesian learning for obtaining
  sparsity of EEG frequency bands based feature vectors in motor imagery
  classification, Int. J. Neural Syst. 27 (02) (2017) 1650032.
http://dx.doi.org/10.1142/S0129065716500325
  doi:10.1142/S0129065716500325.




