Laura Ferrero,Vicente Quiles,Mario Ortiz,Eduardo Iáñez,Ángel Gil-Agudo,José M. Azorín
Brain-computer interface enhanced by virtual reality training for controlling a lower limb exoskeleton
Applied sciences,Engineering,Biomedical Engineering,Control engineering,Robotics
Summary This study explores the use of a brain-computer interface (BCI) based on motor imagery (MI) for the control of a lower limb exoskeleton to aid in motor recovery after a neural injury. The BCI was evaluated in ten able-bodied subjects and two patients with spinal cord injuries. Five able-bodied subjects underwent a virtual reality (VR) training session to accelerate training with the BCI. Results from this group were compared with a control group of five able-bodied subjects, and it was found that the employment of shorter training by VR did not reduce the effectiveness of the BCI and even improved it in some cases. Patients gave positive feedback about the system and were able to handle experimental sessions without reaching high levels of physical and mental exertion. These results are promising for the inclusion of BCI in rehabilitation programs, and future research should investigate the potential of the MI-based BCI system.
Brain-computer interface enhanced by virtual reality training for controlling a lower limb exoskeleton

This study explores the use of a brain-computer interface (BCI) based on motor imagery (MI) for the control of a lower limb exoskeleton to aid in motor recovery after a neural injury. The BCI was evaluated in ten able-bodied subjects and two patients with spinal cord injuries. Five able-bodied subjects underwent a virtual reality (VR) training session to accelerate training with the BCI. Results from this group were compared with a control group of five able-bodied subjects, and it was found that the employment of shorter training by VR did not reduce the effectiveness of the BCI and even improved it in some cases. Patients gave positive feedback about the system and were able to handle experimental sessions without reaching high levels of physical and mental exertion. These results are promising for the inclusion of BCI in rehabilitation programs, and future research should investigate the potential of the MI-based BCI system.

Robotic orthoses and exoskeletons have emerged as wearable devices with the potential to enhance physical performance and assist locomotion. Their inclusion in rehabilitation programs has shown to promote the recovery of motor function, particularly among patients who have experienced a stroke or a spinal cord injury (SCI). These debilitating injuries can significantly impact patients’ quality of life and autonomy, underscoring the importance of robotic exoskeletons in facilitating their recovery and ability to live independently. The interfaces used to control these devices typically rely on a combination of mechanical and electrical instruments. For instance, the H3 exoskeleton (Technaid, Spain) is controlled through commands sent via a smartphone application, whereas the Rex exoskeleton (Rex Bionics, New Zealand) is operated via a joystick. BCIs provide a direct and intuitive mean of communication between subjects and their devices, eliminating the need for external controls., These systems record brain activity and translate it into control commands for an output device. Two types of BCI exist based on the control paradigm used: synchronous and asynchronous. Synchronous BCI rely on external cues to guide the user. Some of them are based on various external stimuli and evoked potentials. Among these, two visually evoked potentials – P300 and steady-state visual evoked potentials (SSVEP),- are the most commonly employed. Another type of synchronous BCI uses oscillatory EEG patterns that subjects modulate using specific mental strategies after a cue. An example of such a BCI is motor imagery (MI), where subjects simulate a given action without physically executing it. Brain activity is only analyzed during predefined time intervals, and a cue indicates the beginning and the end of each interval. In contrast, asynchronous BCI continuously record and analyze brain signals without any temporal constraints. Subjects internally control their brain switches to produce the desired commands for the output device. However, the primary limitation of these BCI relies on its validation. The system either instructs subjects when to switch, or they report themselves changes in their mental procedures. Recent studies have demonstrated that the practice of motor imagery combined with movement-associated feedback may facilitate motor recovery in patients who have suffered neural injuries. This effect is believed to be mediated by the brain’s neuroplasticity, which refers to the nervous system’s ability to adapt and recover by reorganizing its neural pathways. The performance of mental motor imagery with motion feedback has shown to promote this property., In the field of BCI, electroencephalography (EEG) is commonly used to monitor brain activity because of its high temporal resolution, non-invasive recording, and portability. EEG primarily measures the activity of pyramidal neurons that are oriented in the same direction with respect to the cortical surface, allowing one to detect their signals without cancellation. However, EEG has some limitations, as the electrical signal from these neurons is attenuated as it travels through the dura, skull, and scalp before reaching the recording electrode., There are more works in the literature that have worked with EEG based on upper limb MI,,,,, than on lower-limb MI. The main limitation of lower-limb MI is that the leg area of the motor cortex is deeply located, around 1–4 cm from the surface. Therefore, EEG cannot accurately record this activity and it can be highly influenced by activity occurring near the surface of the skull. There are some authors that have tried to discriminate leg MI with respect to other types of MI that can be more easily identified by EEG as left-hand, right-hand and tongue MI and results obtained were promising. However, considering a BCI for rehabilitation, it would be interesting to have a non-intention condition in which subjects do not have to perform MI of another task to pause the system. There are previous studies that have assessed this issue having leg MI versus idle state.,,,,,,,, During idle state subjects must be relaxed. Accordingly, MI acts as the active condition and idle state as the passive one. Nevertheless, the performance metrics of these approaches are worse than the ones that employ upper-limb MI. Although there are many studies based on MI BCI, the closed-loop control of an external device remains briefly addressed in the literature, especially in the case of lower-limb MI. Most of the previous studies performed an offline approach in which the subjects first perform certain brain patterns and once they finish the experiment, data are classified and analyzed. This means that no real-time feedback was provided to the subject during the experimentation, working the BCI in an opened-loop control of the external device. Real-time feedback is crucial for subject’s rehabilitation.,,, Instead of working with MI tasks, some authors have designed experimental approaches for a closed-loop control based on motion intention. Motion intention is a cortical potential produced by the motor cortex just before a movement is performed. In and a patient with spinal cord injury (SCI) walked assisted by an exoskeleton that was triggered by an external operator while their EEG was recorded. Then, motion intention was decoded and used to start/stop the exoskeleton. Moreover, in, four patients with SCI were instructed to attempt to initiate a movement while they were standing and wearing an exoskeleton with their legs blocked. Then, for the real control, patients were asked to perform the same attempt when they wanted to initiate the gait. To ensure safety during the experiment, the participants were only allowed to have complete and unrestricted control over the exoskeleton during specific time intervals, whereas the rest of the time the exoskeleton was blocked. However, the main benefit of MI in comparison with motion intention is that it requires the subjects to mentally repeat a certain movement and they have to be cognitively involved during the duration of the task which promotes motor recovery. Barria et al. combined an MI with visual and/or haptic feedback to enhance the performance of the model, whereas Choi et al. included eye blinking in the control loop. Do et al. developed a BCI based purely on MI of the legs for controlling a treadmill suspended robotic gait orthosis and it was evaluated with an able-bodied individual and a patient with SCI. Rodríguez-Ugarte et al. presented an experimental design for controlling the start of the gait of a robotic exoskeleton based on MI of the gait during determined time intervals. They employed transcranial direct current stimulation (tDCS) for improving the practice of MI and it was evaluated with 12 able-bodied subjects. In our previous studies,, we presented two different BCI based on MI of the gait adding the level of attention of the user in the control paradigm. Therefore, a new command was sent to the exoskeleton only if the attention level to the gait of the user was above a threshold. Whereas the inclusion of this second rule reduced the number of false positives, the sensibility of the model decreased significantly. One of the primary challenges faced by a BCI is its calibration. During the calibration process, the subject performs a series of mental tasks, such as imagining movements and the BCI records the associated brain activity. These data are then used to train the BCI to recognize the patterns of brain activity associated with specific actions. However, this calibration process can be time-consuming and fatiguing for the subjects, particularly for those with impairments who may require shorter session scenarios., Typically, calibration is done at the beginning of each experimental session because of the non-stationary nature of EEG data. However, researchers have been exploring calibration-free approaches using subject-independent techniques or transfer learning to reduce or eliminate the calibration process altogether., However, the effectiveness of these approaches can vary across individuals., Another alternative to reduce the burden of calibration is to improve the quality of the mental tasks performed. In a recent study, participants performed mental imagery tasks while observing an accumulative bar displayed on a screen. The bar increased as the BCI decoded the mental tasks, and subjects were unable to start the experimental session until reaching a certain level of precision. Another challenge in BCI applications based on MI is the variation in the way subjects perform mental tasks, resulting in high variability in success rates. In this study, we tackled both challenges using a virtual reality (VR) system. Participants were immersed in a VR environment where they practiced MI strategies for gait while receiving visual feedback. Then, they used these mental tasks to calibrate a BCI to control a robotic lower-limb exoskeleton without the VR feedback. As participants had already practiced in VR, less data was necessary for the BCI to achieve a high accuracy ratio, reducing the required training time and physical effort. To test the effectiveness of the VR system, we compared the BCI accuracies between subjects who underwent VR training and a control group that did not. The system was tested in closed-loop control with a lower-limb exoskeleton as presented in our previous research, but this time with 12 subjects, including two patients suffering from SCI. This study’s contributions include: - The design and proposal of a VR training system to enhance subjects’ ability to perform MI tasks and speed up the BCI training process. - A comparison of BCI accuracies between subjects that used VR for the BCI training and a control group that did not use it. - Closed-loop experiments for controlling a lower-limb exoskeleton by means of the BCI. - Assessment of these experiments with not only able-bodied subjects but also patients with SCI, analyzing the usability of the BCI application.

Ten able-bodied subjects participated in the study. They had no movement impairment and did not report any known disease. Concerning BCI, they did not have any previous experience. Five participants were randomly included in control group and the other five in the VR group. Furthermore, two patients with SCI were recruited from the National Hospital of Paraplegics in Toledo. They had no prior experience with BCI technology. Further patient’s details are specified in Table 1. Participants wore an EEG cap that recorded their brain-activity and wore a lower-limb robotic exoskeleton that assisted the gait. In addition, participants that were included in VR group used a VR environment. The setup of the experiment can be seen in Figure 1.

Three distinct experiments were conducted, as illustrated in Figure 2. In the first experiment, five participants underwent a conventional calibration phase with the BCI. The calibration was followed by a closed-loop evaluation, where the exoskeleton was controlled by their thoughts resulting in motion feedback. They participated in five sessions scheduled in five different days, each of which consisted of a calibration and a closed-loop phase. Experiment 2 involved the VR system, in which five subjects participated in five experimental sessions too. Each session followed a similar format to the first experiment, but included a practice of mental tasks during VR immersion with visual feedback. The calibration phase was then shortened to half of that in experiment 1 before moving onto the closed-loop phase. Finally, experiment 3 involved two patients with incomplete SCI who underwent the same experimental sessions as in experiment 2, but with a reduced duration of VR immersion. However, in this experiment, the first of the five sessions did not involve closed-loop control phase because patients needed more time to become familiar with the system.

In Figure 2, we can see that experiment 1 involved five able-bodied participants who took part in five experimental sessions. Each session consisted of a calibration phase, during which the participants performed 22 trials. The sequence of each trial can be seen in Figure 3A. The trial began with a 15-s period in which the participants were not required to perform any mental task. This initial data was utilized to adjust the algorithms that were used to clean the EEG signals. After this, an acoustic cue signaled the start of the first mental task, which involved the participants relaxing in an idle state. Following this, another acoustic cue indicated the start of the MI period, during which the participants were instructed to focus on imagining the movement of their legs as if they were walking (kinesthetic imagination). Then, another cue signaled the transition to the regressive count task, during which the participants had to mentally perform a series of subtractions (e.g., 900-7, 893, 886, 879, etc.). Finally, a cue specified the start of the second idle state period. It is important to note that the data obtained during the regressive count period was not used for this BCI and was discarded. Therefore, only two classes were considered: ‘Idle state’ and ‘MI of the gait’. As reported by Ferrero et al., in the experimental setup, participants were assigned to perform two different types of trials: standing still with the exoskeleton (11 trials) and walking with the exoskeleton (11 trials). The exoskeleton was programmed to initiate and terminate gait based on pre-established commands, depending on whether the trial was in static or motion. Notably, the sequence of tasks was identical for all trials (as depicted in Figure 3A), as this was done to facilitate the creation of a dual-state BCI (illustrated in Figure 4), which was then utilized in the closed-loop control phase. A BCI based on motor imagery usually decodes two mental states to command the exoskeleton: ‘Idle state’ versus ‘MI’. However, to ensure the accuracy of the BCI, the training data used to calibrate the BCI must be obtained under the same conditions as during the closed-loop phase. This means that the idle state and MI mental classes should be collected not only when the exoskeleton is static but also when it is in motion. To achieve this, four different events were collected during training trials: static idle state, idle state in motion, static MI, and MI in motion. These four classes were then used to create two different classifier models: Static (‘Static idle state’ versus ‘Static MI of the gait’) and Motion (‘Idle state in motion’ versus ‘MI of the gait in motion’). In the closed-loop control phase, the initiation and cessation of the exoskeleton gait were controlled by the subject’s mental practices and the BCI decoding system. During this phase, subjects performed five trials as described in Figure 3Bwhich included the same mental tasks as in the calibration phase but without the regressive count event. The models trained during the calibration phase, namely Static and Motion, were utilized for the control process (see Figure 4). As a dual-state machine, the Static model was used to keep the exoskeleton standing still and to detect the starting of the gait. However, once the exoskeleton initiated the gait, the BCI control model switched to Motion model, which was used to maintain the exoskeleton in motion until the desire to stop was detected. This would prompt a change of the model to Static again. Because the classes used in the model were collected in the same noise conditions, it was ensured that the differences between the classes were solely based on the mental tasks.

In experiment 2, the study included five able-bodied participants who were engaged in five experimental sessions. These sessions were similar to experiment 1. However, they also involved a prior training phase which included VR immersion. In this experiment, a shorter calibration phase with only 12 trials (six in static and six in motion) was used. During the VR immersion phase, the participants were placed in a virtual corridor inside a spacecraft in a first-person view, as depicted in Figure 5. They completed ten trials under similar conditions as the calibration phase (see Figure 3A). For five of these trials, the participant’s avatar walked through the corridor, and for the other five, it stood still, providing visual feedback. It should be noted that the movement of the avatar was predefined, and did not depend on the participants' mental practices. The closed-loop phase was identical to experiment 1.

Once the experimental setup was evaluated with able-bodied subjects, it was tested with patients with SCI. Accordingly, they participated in five sessions in different days and each session included three phases: VR, calibration, and closed-loop control of an exoskeleton. The sequence of mental tasks to perform was identical to Figure 3. The only difference with respect to able-bodied participants was that VR sessions were shorter. They performed 6 trials in the VR environment: three with the avatar being static and three with the avatar walking through the corridor. Then, they performed 12 trials for the calibration. And for closed-loop control of an exoskeleton, participants did seven trials. However, only five were in real control and considered for metrics assessment of the BCI, in the other two the BCI was manually forced to work perfectly as if they did the mental practices well. This strategy was employed to prevent them from getting frustrated. During each session, we measured the level of exertion that patients perceived with the scale Borg Rating of Perceived Exertion Scale. This range goes from 6 (“not effort at all”) to 20 (“absolute maximal effort, exhaustion”), and it was used as an indicator to terminate the experimental session if the level perceived was close to the maximum. In addition, after the last experimental session, patients filled two questionnaires: NASA Task Load Index (NASA-TLX) that estimated the perceived mental workload; and Quebec User Evaluation of Satisfaction with Assistive Technology (QUEST 20) that evaluated the satisfaction with the exoskeleton.

This section compares results obtained in calibration phase between control and VR group. Figure 6 shows the average accuracy () for each session and subject. Static and trials in motion were treated independently because they were produced under different conditions. Figure 6A shows results of cross-validation leave-one-out with static trials from all ten healthy individuals. S1–S5 represent the subjects that were included in the control group from experiment 1 and V1-V5 the subjects that were included in the VR group from experiment 2. Regarding trials in motion, Figure 6B shows their results. The average (avg.±std.) of static trials from control group for each experimental session was: 70.214.2%, 69.614.5%, 72.512.9%, 71.914.6%, 66.415.1%. For trials in movement, performances per session were: 53.38.0%, 57.05.4%, 63.512.4%, 64.914.1%, 63.68.0%. The average accuracy of VR group per session with static trials was: 69.512.4%, 65.813.0%, 63.816.5%, 70.216.8%, 65.417.1%. The average of trials in movement from VR per session was: 61.312.0%, 64.68.7%, 71.012.2%, 61.614.1%, 65.116.7%. To determine if there were any noteworthy distinctions between the VR group and the control group, a statistical analysis was conducted. However, the variability introduced by the grouping was not the only factor that could affect the results; the experimental sessions could also have an impact. Two separate analyses were conducted for static and trials in motion (further details are included in quantification and statistical analysis in STAR Methods). • Static trials: No significant differences were found neither among sessions nor among groups. • Trials in motion: The test showed that significant superior results were seen for VR group. Differences among sessions were significant. In addition, the interaction between group and sessions was significant. It means the evolution of accuracy through sessions varies from VR to control group. Figure 7 shows boxplot of accuracy values for each session and group. It can be observed that VR group outperforms control group in all sessions but session 4. Figure 7 Boxplot of accuracies obtained in trials in movement for each session and each group Subjects who participated in training with a VR environment and subjects who did not (control). %MI is the accuracy of the model that goes from 0 to 100. Figure 8 shows boxplots of accuracies obtained for all trials of each subject. It seemed that the performance of some of them was really different from the rest. Thus, a statistical test was conducted to see if these differences were relevant (further details are included in quantification and statistical analysis in STAR Methods). • Static trials: The test showed significant differences among subjects. • Trials in motion: The test showed significant differences among subjects too. Results could be influenced by the fact that subjects had a different training, and it was previously proven that VR group tended to have higher results. In the control group, there were three sub-groups of subjects that showed a significant different behavior: S1 and S2; S3 and S4; and S5. A similar trend was found in the VR group with three sub-groups: V1 and V3; V2 and V4; V5. Intending to assess the evolution of performance through sessions, metrics were normalized with respect to the first session. This way, it can be studied how the accuracy evolved. New values of accuracy were computed as: is the accuracy obtained for a trial j in an experimental session i, and is the average accuracy of all trials of the first session. This value ranges from −1 to 1. Figure 9 shows the average that goes from −1 to 1. Statistical analysis was repeated with this new metric for static and trials in motion (further details are included in quantification and statistical analysis in STAR Methods). • Static trials: No differences were found among sessions or the VR and control group. • Motion trials: Relevant differences were found among sessions and the interaction between sessions and groups was significant. Moreover, the results from control group were substantially better than the ones of the VR group. Figure 10 shows the distribution of for each subject. Metrics were normalized with respect to the first session. However, after performing a statistical analysis, significant differences were still found in static and motion trials among subjects (further details are included in quantification and statistical analysis in STAR Methods). To investigate potential differences in neural activity between groups and individuals, power spectral density (PSD) was computed in alpha and beta frequency ranges. These frequency ranges have been associated with motor processes in EEG literature. EEG signals were pre-processed according to method details in STAR Methods, and then subjected to independent component analysis (ICA) as a denoising technique. As ICA cannot be used in real-time, it was only used for offline analysis. The ratio of PSD during MI periods and idle state periods was computed using the formula: This calculation was performed for all electrodes and averaged across all trials and sessions for each subject. The resulting values were then represented as topographical maps of the scalp, providing a spatially-detailed representation of changes in neural activity. The resulting maps are displayed in Figure 11 and 12. The spectral maps exhibited significant variation among subjects, but a common decrease in power was observed in motor areas, particularly in subjects with high accuracy such as S3, S5, and V4. These observations align with previous research in the field. During the cognitive practice of MI, there appears to be a decrease in relative power of EEG signals in frequencies related to motor function, which is then followed by an increase in power. This phenomenon is commonly referred to as event-related desynchronization/synchronization (ERD/ERS) 42. No significant differences were observed between the groups, which is advantageous for VR group as they exhibited comparable patterns with less training.

This section presents the findings from the closed-loop control phase of experiment 1 and 2, as observed in the five participants in the control group and the five participants in the VR group. Throughout this phase, the exoskeleton was directed by each subject’s cognitive techniques. The results from the control group participants are presented in Tables 2, 3, 4, 5, and 6,whereas those from the VR group participants are detailed in Tables 7, 8, 9, 10 and 11. Table 12 presents a detailed comparison of the average results for both groups. Metrics are further explained in Evaluation metrics in STAR Methods. In both instances, Motion model demonstrated superior performance in comparison to Static model, indicating that the model controlling continuous walking or stopping was more effective. Analyzing the outcome of the experiments conducted with , , and per epoch, both groups exhibited a similar pattern of behavior. However, in keeping with the trend observed in the calibration trials, the performance of Motion model in the VR group far surpassed that of the control group.

Experiment 3 consisted of evaluating the proposed BCI and training approach with patients with SCI. Moreover, this research evaluated the acceptance and usability perceived from them. QUEST questionnaire evaluated their satisfaction with the lower-limb exoskeleton H3 as an assistive technology. Their responses are shown in Table 13. Patient P1 was in general more satisfied with the exoskeleton than patient P2. Both agreed that it was not very comfortable, and P2 found that the adjustment was not easy. Table 14 shows their responses to NASA-TLX questionnaire that assessed the workload perceived. They felt they had to work hard physically and mentally to reach desired results, but they were satisfied with their performance. In addition, Borg scale was employed to measure the effort. It was measured after preparing all equipment, at the end of the VR phase, after an initial walk with the exoskeleton without the BCI, so they could get used to it, at the end of the calibration phase and at the end of the phase in which they controlled the exoskeleton with brain tasks. Results from this last scale are shown in Table 15. During each session, the level of exertion increased progressively, but patients never experienced values of exertion close to their own maximum. The performance of the two patients during the calibration phase (see Figure 13) is within the range of able-bodied individuals (see Figure 6). Because there are significant differences among subjects, it is difficult to say if the performance of patients is in average lower or higher than able-bodied ones. Table 16 and 17show results obtained in closed-loop trials. Contrary to able-bodied individuals, Motion model was less efficient than static one.

In this study, the performance of two groups of subjects was compared. The control group underwent a calibration phase consisting of 11 static and 11 movement trials, whereas the VR group underwent an additional previous phase in a VR environment, followed by a calibration phase consisting of six static and six movement trials. It was observed that the VR group experienced less physical exertion because of the reduced number of walking trials with the exoskeleton. Significant differences were observed among the subjects, with the VR group demonstrating superior performance in trials involving movement. However, no significant differences were observed in static trials. It is noteworthy that even if the performance of the two groups was similar, the VR approach could offer an alternative method of training that requires less physical effort. Furthermore, when the training was divided into VR and calibration phases, it was perceived as less monotonous, which can be advantageous in keeping the subjects motivated and focused. The evolution of performance through sessions was only found significant in trials in movement. Accuracy values were significantly different from last session to the first one, but accuracy growth was different among subjects. This growth was significantly different from subjects in VR and controls. The global trend was similar for the first three sessions, but in the fourth one, VR subjects got lower accuracy than in the previous session.

This study employed a dual-state machine for the BCI, comprising two distinct models. Static model was activated when participants were stationary, whereas Motion model was activated when participants walked with the assistance of an exoskeleton (as illustrated in Figure 4). The performance of the two models varied significantly, with Motion model exhibiting superior performance in both participant groups. This was primarily attributed to the higher FPR of Static model, which made it challenging for participants to remain relaxed when they had control over the exoskeleton, an experience not encountered during the calibration phase. The , , and indicators evaluated the efficiency of the BCI as a whole, independent of the performance of each model. Both groups displayed similar behavior when considering these indicators. However, focusing on the performance of each model, the weighted discriminator proved to be the most effective metric. Although the performance of Static model was similar across groups, Motion model of the VR group outperformed that of the control group. These findings align with those observed in the calibration phase (refer to section 4.1). As previously discussed, there were notable variations in performance across participants. Within the control group, S5 exhibited the best results, whereas S2 demonstrated the lowest performance, aligning with their respective performance in the calibration phase. In the VR group, V2 achieved the highest outcomes, whereas V5 showed the lowest. Of interest, during the calibration phase, it was V4 who demonstrated the highest performance, but the values of underwent substantial changes from calibration to closed-loop phases. Various factors can influence the system’s performance, such as the cognitive state and concentration level of the subject., Within the control group, three subjects did not miss any trials in all sessions, whereas in the VR group, all participants except V2 did not miss any trials in all sessions. These metrics can be interpreted as 88% and 98% of trials where at least one activation command was issued during motor imagery. These results can be compared to the 84.4% accuracy reported by, whose approach was based on motion intention, and the exoskeleton was blocked when subjects were required to relax, giving them control over it only during specific time periods. In the research presented by, the average accuracy reached 84.45.43% when starting the gait on ten able-bodied subjects. They defined a model that compared gait MI with sit MI so subjects could start the gait with the exoskeleton or could sit. This metric can be compared with our accuracy of commands in Static model. Their approach outperforms our proposed BCI. A possible explanation could be that BCI models perform best when different MI patterns are being compared., However, it was crucial in our design to include a non-intention condition. reported an accuracy of 73.4% in seven participants. Because the exoskeleton was only free during periods of MI, these metrics are in accordance with our Motion model that showed an average accuracy of 49.8632.13% in control group and 66.814.98% in VR group. In addition, our approach was completely free for the user to transition among tasks. In our previous studies,, MI and attention paradigms were combined to control an exoskeleton. In, %Commands was 56.77% and in it was 64.08%. Results obtained in this paper are within the same range, but in the VR group less information was employed to train the classifier which supposes a similar performance with a lower protocol time.

Results from calibration phase of P1 showed a positive trend throughout sessions. This trend was evident in static and trials in motion. P2 did not demonstrate the same learning effect. It is important to note that whereas both patients suffered an incomplete SCI, the injury of P2 affected motor function to a greater extent. P2 may need more practice with the system before reaching an acceptable level of performance. The ratio of missing trials was 0 for P1 and 0.05 for P1. Therefore, P1 could activate the exoskeleton in all the trials and P2 only missed one trial in last session. Results are in line with. They evaluated their BCI with a patient with SCI and it was only focused on activation of the exoskeleton which corresponds to our Static model. On the other hand,, defined a closed-loop scenario in which an SCI patient was asked to stop the gait. They obtained 99.31% correct stopping commands. It is difficult to compare this study with our research because in ours, subjects had to stop the exoskeleton in a fixed short period of time instead of having several attempts to do it. evaluated their approach of combining visual/haptic feedback, and MI with five post-stroke patients. They got 50.7 and 68% of successful activation attempts which could be comparable to our TPR in static. However, although patients who suffered an SCI do not have cognitive affectation, patients who suffered a stroke have brain damage, which makes it difficult to do a comparison. TPR in static was 77.71% for patient P1 and 76.67% for P2. These metrics are the number of successful activations of the exoskeleton out of all the attempts. , used motion intention to start the movement and then, the system kept it. They both were evaluated with four SCI and results from were 79%, 93.10%, 87.10% and 68.81% for each patient. Again, these metrics can be confronted to our TPR in static. reported an average accuracy of 77.6%. However, metrics were computed with different rules as a trial was considered successful if at least an activation command was sent to the exoskeleton. In our results, P1 got a 100% of trials in which at least a command was sent to the exoskeleton (0 missing trials), and P2 got 100% in all sessions, but last one with 80%. The proposed BCI was proven to be at least as effective as previous approaches presented in the literature with more restricting assessing metrics. Furthermore, the BCI implemented in this study provided complete and unrestricted control for patients to initiate, maintain, stop, and remain stationary during gait without any predefined time limitations imposed by the experimental protocol or periods of no control over the exoskeleton. This was achieved by designing a dual-state machine control BCI that relied on two distinct models to initiate and terminate exoskeleton movement, resulting in a more realistic and practical approach to the real-time control of an exoskeleton.

This research has some limitations, especially regarding the dataset. All able-bodied participants shared common features. They were in their 20s, presented right-dominance laterality, and did not have previous experience with a BCI. Nevertheless, there were significant differences among subjects in terms of BCI performance. Thus, it was difficult to draw conclusions regarding performance. For future studies, it would be interesting that each subject tries both experimental conditions with a random order and both spaced in time. Half of them would be first included in control group and then in VR, and the opposite for the other half. This way, differences among subjects would be mitigated. Providing the great dependence on subjects, another limitation was the size of the dataset. Although more subjects participated in this study than for most of BCI applications, the database was not vastly large to represent all possible usage outcomes. Consequently, it was difficult to demonstrate which would be the average efficiency of the system. Future research should consider extending the sample size to have a better representation of the whole population. On the other hand, for training with the VR environment, subjects wore the exoskeleton, and it was blocked in a standing straight position. It involved less physical exertion than walking with the exoskeleton, but it still required some effort, especially for patients. Looking forward, further attempts could evaluate a training phase with VR devices in which participants remain seated.

Overall, current research compared two training alternatives for an MI based BCI for controlling a lower-limb exoskeleton. One alternative consisted of conventional calibration training with the BCI and the exoskeleton; and the other consisted of a pre-calibration session with a VR environment combined with a shorter conventional calibration phase. Results demonstrate that the employment of a shorter calibration phase which involves less information for the model did not worsen the effectiveness of the BCI. In fact, the performance of the Motion model improved. Moreover, the BCI was evaluated for controlling a lower-limb exoskeleton in ten healthy subjects and two patients with SCI. Usability and acceptance were evaluated in patients. Results were promising for a lower-limb MI based BCI system. Consequently, future research could continue to explore the inclusion of this type of systems with robotic devices in rehabilitation programs.



Further information and requests for resources should be directed to and will be fulfilled by the lead contact, José M. Azorín (jm.azorin@umh.es).

This study did not generate new unique reagents.

Ten able-bodied subjects participated in the study (mean age, 23.5 ± 2.0). Five females and five males. They had no movement impairment and did not report any known disease. They were informed about the experiments and signed an informed consent form in accordance with the Helsinki declaration. All procedures were approved by the Responsible Research Office of Miguel Hernández, University of Elche Spain (DIS.JAP.03.18, 22/01/2019). Five participants were randomly included in Control group and the other five in the VR group. In addition, two patients with spinal cord injury (SCI) were recruited from the National Hospital of Paraplegics in Toledo. They also signed an informed consent form in accordance with the Helsinki Declaration and had no prior experience with BCI technology. Further patient’s details are specified in Table 1.

EEG signals were recorded with Starstim R32 (Neuroelectrics, Spain) at 500 Hz. The placement of 27 wet electrodes followed the distribution of the international 10-10 system, with electrode positions including F3, FZ, F4, FC5, FC3, FC1, FCZ, FC2, FC4, FC6, C5, C3, C1, CZ, C2, C4, C6, CP5, CP3, CP1, CPZ, CP2, CP4, CP6, P3, PZ, P4. Ground and reference electrodes were positioned on the right ear lobe. NIC v2.0.5c software (Neuroelectrics, Spain) was used to record EEG in the laptop. All participants wore H3 exoskeleton (Tecnhnaid, Spain). It is a powered hip-knee-ankle exoskeleton that emulates the human walking. Participants were provided with crutches or a trolley to ensure standing and walking stability. To prevent any risk of falls, a technician also held the exoskeleton from the back. High-level control commands were transmitted from the computer via Bluetooth to initiate and stop the gait at a constant speed. Virtual reality equipment, including an HTC VIVE headset (HTC, Taiwan) with a resolution of 2160 x 1200 (1080 x 1200 per eye) and a refresh rate of 90 Hz, two base stations to track the headset’s location, Steam software (Valve, United States), and a self-developed environment created with Unity engine (Unity Technologies, United States) were used in the study. The equipment setup is illustrated in Figure 1.

This section describes the analysis performed to decode mental tasks from EEG data. MATLAB 2018a was used to connect through TCP/IP with NIC v2.0.5c software and get EEG data, analyze it and send control commands to the exoskeleton. While the exoskeleton was controlled in closed-loop, a decision was taken every 0.5 s. This means that every 0.5 s, the BCI could decide to stay static, start walking, keep on walking, or stop the gait. The epochs of analysis were 1 s length with a shifting of 0.5s. Therefore, once 0.5 s of current data were recorded, epochs of 1s had to be pre-processed, processed, classified, and translated into a control instruction before new 0.5 s of data were available. The analysis of trials in opened-loop control, such as VR and calibration phases, was performed simulating a real-time scenario (pseudo-online analysis). Consequently, although the analysis was performed afterward, it was done with epochs of data of 1 s and 0.5 s of overlapping. Figure S1 shows the overall schema of the BCI.

The first step of the BCI consisted of increasing the signal to noise ratio. Epochs of 1 s of EEG were filtered with a Notch filter at 50 Hz that mitigated the contribution of the power line, and a high-pass filter at 0.1 Hz for reducing DC offset. The Notch filter was directly applied in the NIC recording software and the high-pass filter was applied with state-variables Butterworth function in MATLAB.

The following step focused on extracting signal features that could be used to discriminate between different mental patterns. Our BCI used filter-bank common spatial patterns (FBCSP). Firstly, signals were band-pass filtered into different frequency bands: 5–10, 10–15, 15–20, 20–25 Hz. Then, common spatial patterns (CSP) were extracted from each band. CSP estimated a spatial filter that maximized the discriminability between two classes. ∼N·T and ∼N·T are EEG data from two different brain patterns. N is the number of channels and T is the number of samples. In CSP algorithm, their covariance matrices were computed as: Covariance matrices were calculated independently for each trial and then, averaged. Afterwards, they were combined and factorized: is the diagonal matrix of eigenvalues and is a matrix of eigenvectors. Subsequently, results from previous factorization were used to transform each averaged covariance matrix: As a result of Equation 6, and had the same matrix of eigenvectors and the addition of the two matrices of eigenvalues resulted in the identify matrix. and were employed to estimate the spatial transformation matrix, . The original data was transformed with . Resulting had the same dimensions as the original data, N·T, but new channels were ordered based on their discriminative power. Top and last channels were more discriminative in terms of variance and middle ones were less discriminative. For dimensionality reduction, only m = 3 first and last discriminant rows of Z were selected as features, In addition, the variance of each dimension of was calculated and log-normalized.

As explained in the previous subsection, a vector of features was obtained for each epoch of data. Then, a linear discriminant analysis (LDA) was used as the classifier. It was trained with epochs of class ‘Idle state’ and ‘MI of gait’, and it predicted these two classes. Two models were created independently for the static and motion trials.

Trials from calibration phase in the three experiments were evaluated with leave-one-out cross-validation. Moreover, trials in static and trials in motion were assessed independently. In this study, two approaches were compared, control and VR group (experiment 1 and 2). Therefore, for having a fair comparison, the classifier was trained with the same number of trials. For each subject and session. • If the subject was in control group: o Leave-one-out cross-validation with last six training trials in static. o Leave-one-out cross-validation with last six training trials in motion. • If the subject was in VR group: o Leave-one-out cross-validation leave-one-out with all six training trials in static. o Leave-one-out cross-validation with all six training trials in motion. For both conditions, one trial was selected for evaluation while the remaining five trials were used for training the classifier model. This process was repeated for each of the six trials, so that each trial was used for evaluation once. The accuracy of the BCI system was then calculated as the average of the six cross-validation iterations for both the static and motion conditions.

Two LDA classifiers were trained using the data from the calibration phase for the static and motion conditions, respectively. These classifiers were denoted as Static and Motion. Subsequently, based on the status of the exoskeleton, either stationary or in motion, one of the classifiers was used during each epoch and it predicted either ‘Idle state’ or ‘MI of the gait’. More information about this process is provided in section 2.3 and experiment 1. A depiction of this dual-state machine BCI can be seen in Figure 4.

There was an additional step in the BCI for the closed-loop control of an exoskeleton that consisted of translating predictions from the classifier into control instructions. The classifier provided predictions with a value of 1 for epochs labeled as ‘MI of the gait’ and 0 for epochs labeled as ‘Idle state’. To minimize the number of false positives and false negatives, these predictions were smoothed using a moving mean of 4 s. The resulting values ranged from 0 to 1, with values above a‘activation threshold’ triggering a command to initiate the gait. Additionally, during movement, values below the ‘deactivation threshold’ would trigger a command to stop the exoskeleton’s motion. To determine the appropriate thresholds for activation and deactivation, calibration data was utilized. Specifically, the results from a leave-one-out cross-validation with six static trials were used to determine the ‘activation threshold’, while the results from a leave-one-out cross-validation with six motion trials were used to determine the ‘deactivation threshold’. To optimize the ‘activation threshold, predictions from six static trials were utilized. These predictions were first smoothed using a moving mean of 4 s. To identify the optimal activation threshold, three plateaus were identified from each static trial: the first idle state, MI, and second idle state periods. A plateau was defined as consecutive points with a slope near 0, ranging between −0.05 and 0.05. Figure S2 demonstrates an example of a trial and the three plateaus that were identified. In cases where more than one plateau was found for each task, the highest one was selected. Once the plateaus were identified, the intermediate point between the plateau of the first idle state and the plateau of MI was computed and defined as the single trial 'activation threshold' as can be seen in Figure S2. This process was repeated for all six static trials. Outliers were subsequently removed, and the remaining resting values were averaged to identify a reliable activation threshold that could be utilized in the closed-loop control of the exoskeleton. Similarly, the process was repeated using results from motion trials to define the 'deactivation threshold'. In this case, the threshold was defined as the intermediate point between the plateau of MI and the plateau of the second idle state period. Once again, outliers were removed, and resting values were averaged to obtain a reliable deactivation threshold that could be utilized in the closed-loop control of the exoskeleton. As the last step, some rules were defined to avoid overfitting and to correlate both thresholds: • ‘Activation threshold’ could not be lower than the average prediction value during first idle state period (in static trials). • ‘Deactivation threshold’ could not be higher than the average prediction value during MI period (in trials in motion). • ‘Activation threshold’ could not be higher than the average prediction value during MI period (in trials in motion).

The accuracy was computed as the percentage of epochs with correct predictions (%MI). This value was estimated for each trial without considering the initial task employed for the convergence of the algorithms.

The following metrics were computed for evaluation. - %MI: percentage of epochs with correct predictions. - %Commands: percentage of epochs with correct commands issued. - %Feedback EXO: percentage of epochs with correct feedback from the exoskeleton. - Missing trials: percentage of trials in which the exoskeleton never activated during MI period. - Static model o TPR (true positive ratio): ratio of how many activation commands were issued during MI period out of how many times subjects were static. o FPR (false positive ratio): ratio of how many activation commands were issued during idle state period of how many times subjects were static. o Accuracy commands: percentage of correct activation commands issued out of all activation commands. o WD (Weighted discriminator): this metric was introduced in, it ranges from −1 to 1 and it is calculated as: - Motion model o TPR: ratio of how many deactivation commands were issued during idle state period out of how many times subjects were moving. o FPR: ratio of how many deactivation commands were issued during MI period of how many times subjects were moving. o Accuracy commands: percentage of correct deactivation commands issued out of all deactivation commands. o WD (Weighted discriminator): it is calculated as detailed above but with metrics corresponding to Motion model.

To determine if there were any noteworthy distinctions between the VR group and the control group in calibration data, a statistical analysis was conducted with Rstudio 1.1.441 and R programming language 4.0.2. The variability introduced by the grouping was not the only factor that could affect the results; the experimental sessions could also have an impact. To account for this, a two-way ANOVA test was performed with accuracy (%MI) as the dependent variable and group and session as independent variables, accuracy ∼ group·session. In order to obtain reliable results from the ANOVA test, the dataset must satisfy three assumptions: normality, equality of variances, and independence. Two separate analyses were conducted for static and trials in motion. Saphiro test was employed to test normality with saphiro_test() function and data was organized by group and session variable with group_by(). Equality of variances was evaluated with Barlett test and bartlett.test() function. • Static trials: Sincedata did not follow a normal distribution, Kruskal-Wallis test was employed instead of ANOVA with kruskal.test() function. No significant differences were found neither among sessions nor among groups (pvalue>0.05). • Trials in motion: All assumptions were not met, so Kruskal-Wallis test was conducted. It showed that significant superior results were seen for VR group (pvalue<0.05). Differences among sessions were significant (p value<0.05). In addition, the interaction between group and sessions was significant. It means the evolution of accuracy through sessions varies from VR to control group. Furthermore, to determine the significance of inter-subject differences in accuracy, a separate ANOVA test was performed for each trial type using accuracy as the dependent variable and subjects as the independent variable, accuracy ∼ subjects. • Static trials: Data satisfied all requirements and the results of the ANOVA test showed significant differences among subjects (p value<0.05). The test was performed with aov(). • Trials in motion: Requirements were also met and the results from the ANOVA test showed significant differences among subjects (p value<0.05) too. Results could be influenced by the fact that subjects had a different training, and it was previously proven that VR group tended to have higher results. Therefore, two independent ANOVA tests were conducted for the control and VR subjects. In the control group, there were three sub-groups of subjects that showed a significant different behavior: S1 and S2; S3 and S4; and S5. A similar trend was found in the VR group with three sub-groups: V1 and V3; V2 and V4; V5. Intending to assess the evolution of performance through sessions, metrics were normalized with respect to the first session. This way, it can be studied how the accuracy evolved. New values of accuracy were computed as: is the accuracy obtained for a trial j in an experimental session i, and is the average accuracy of all trials of the first session. This value ranges from −1 to 1. Figure 9 shows the average that goes from −1 to 1. The two-way ANOVA accuracy∼ group·session was repeated with this new metric for static and trials in motion. • Static trials: Data did follow a normal distribution, but the variance was not equal for all groups. Thus, Kruskal-Wallis test was conducted and no differences were found among sessions or the VR and control group (p value>0.05). • Motion trials: As the requisite of equal variances was violated, Kruskal-Wallis test was employed. It found relevant differences among sessions and the interaction between sessions and groups was significant. Moreover, the results from control group were substantially better than the ones of the VR group (p value<0.05). Lastly, an assessment was made of inter-subject differences in normalized accuracy, . As the distribution of normalized accuracy was non-normal, a Kruskal-Wallis test was utilized. Significant differences were observed among subjects in both static and motion trials (p value<0.05).